The ChatGPT o1-preview is thinking less and becoming dumber than before

Potential Performance Degradation in o1-preview: A Comparative Analysis

I’ve noticed some intriguing changes in the o1-preview model’s performance lately, and I’d like to share my observations with the community. I’m curious if others have experienced similar patterns.

Key Observations

Response Time and Quality

In recent interactions (February 2025), the model consistently produces responses within approximately 8 seconds - significantly faster than its previous behavior. While faster responses might seem beneficial, I’ve observed a concerning trade-off in quality and quantity (in term of output tokens).

Comparative Cases

Case Study 1: Reasoning Depth

  • December 2024: Reference Chat

    • Demonstrated 25-second deliberation
    • Provided well-structured rationale
  • February 2025: Reference Chat

    • 8-second reasoning period with unhelpful info

Case Study 2: Problem-Solving Capability

  • November 2024: Reference Chat

    • 4+ minutes of comprehensive analysis on first shot
    • Detailed problem-solving process with iterative improvements
  • February 2025: Reference Chat

    • 8-second reasoning time and
    • Demonstrably incorrect solutions

Community Input Request

To validate these observations, I encourage you to conduct your own experiments:

  1. Retrieve your o1-Preview interactions from 2024
  2. Submit identical prompts to the current model
  3. Compare response quality, reasoning depth, and processing time

Could this be related to the transition toward the o3-mini series, or the rollout of ChatGPT Pro? I’d appreciate your insights and experiences.


Note: This isn’t meant to be conspiratorial - just a genuine attempt to understand these observed changes in model behavior.

1 Like