Has There Been A Recent Decrease In GPT-4 Quality?

As OpenAI does not respond, I asked their IA :innocent:

User: Between the March 2023 version and the May 2023 version, the features of chatGPT-4 have drastically evolved. The model is noticeably faster, but it makes gross reasoning errors and quickly loses track of the conversation. It regularly forgets instructions and its understanding is less subtle. At the same time, the number of subscribers has significantly increased, straining the system. The limit on the number of messages has been reduced. What could be the causes of these developments? Give me 5 reasoned propositions.

ChatGPT:

  1. Modification of the learning algorithm: The researchers may have modified the learning algorithm between these two versions in order to improve the speed of the model. This could have led to undesired side effects such as reasoning errors and less subtle understanding. For instance, they could have reduced the number of layers in the model or the number of neurons per layer, which could make the model faster, but less accurate and less capable of maintaining the conversation’s context.
  2. Infrastructure issues: The increase in the number of subscribers might have strained the hardware and software infrastructures that host and execute the model. This could lead to reductions in performance, memory management problems, and thus a reduction in the number of possible messages.
  3. Changes in the training data: If the training data used for the model changed between March and May, this could also explain some of the observed modifications. For instance, if the training data was noisier, less varied, or contained more errors, this could be reflected in the behavior of the model.
  4. Optimization for specific metrics: The developers might have optimized the model for specific metrics that favor speed over accuracy or consistency. For example, they could have prioritized minimizing response time over the accuracy of the responses.
  5. Modifications to the model’s architecture: Developers might have modified the architecture of GPT-4 between these two versions. For instance, they could have changed how attention is calculated, or modified the type of layers used. These modifications could improve certain aspects of the model (like speed), but degrade other aspects (like accuracy or the ability to follow the context of a conversation).

User: Now give me 5 reasons why these modifications were made.

ChatGPT:

  1. Increase operational efficiency: Modifications that make the model faster can reduce operating costs. This could be particularly relevant if the number of users of the model has increased, as you indicated.
  2. Meet user demand: Users might request faster responses, even if this might lead to a slight decrease in the quality of responses. Indeed, in many use cases, response speed can be more important than accuracy.
  3. Experimentation and learning: Developers might make modifications to experiment and learn how they affect the model. For instance, they may want to understand how modifications in the learning algorithm or the model’s architecture can affect speed and accuracy.
  4. Improve scalability: With an increase in the number of subscribers, it’s important that the model can be deployed at a large scale efficiently. The modifications might have been made in order to improve the system’s scalability.
  5. Respond to resource constraints: Hardware constraints, like storage capacity or available bandwidth, may require modifications to the model. For example, if the model generates fewer texts, this could reduce the load on the servers and allow more users to use it simultaneously.

User: At the same time, there has been no communication about what has been done and why, the quality of responses has drastically reduced, but the price of the subscription is still the same. What to think about this situation?

ChatGPT:

  1. Transparency: Transparency is an important principle in the field of AI. If significant modifications are made to a service, especially those that directly affect the user experience, it is generally good to proactively communicate them. This can include an explanation about what was done and why.
  2. Quality of service: If the quality of the service has decreased, that is a legitimate concern. Users generally expect the service to be maintained at a certain quality level, if not improved, especially if the cost remains the same.
  3. Value for the user: Even if the price remains the same, if the quality of the service has decreased, it could mean that users are getting less value for their money. This can raise concerns and possibly lead users to look for alternatives.
  4. Expectation management: If the changes have led to a drop in model performance without prior communication, it can create disappointment and loss of trust. It’s important to manage user expectations to maintain a good relationship with them.
  5. Opportunity for feedback: If users are not satisfied, they should have the opportunity to provide feedback to the company. This feedback can help the company understand user concerns and make improvements.

User: Why has there been no communication on this subject?

ChatGPT:

The lack of communication could be due to a number of factors:

  1. Prioritization: The company might be focusing on internal operations and modifications and might not have prioritized communicating about these changes to its user base.
  2. Lack of clarity: The company might be unsure about the impact or permanence of these changes, and therefore hesitant to communicate about them until there is more certainty.
  3. Oversight: It could be a simple oversight or error, where the importance of communication wasn’t fully recognized or was underestimated.
  4. Strategic decision: It might be a strategic decision where the company feels that discussing these changes might lead to more confusion or dissatisfaction among users.
  5. Concerns about perception: The company might be concerned about how this news would be received, particularly if the changes are perceived as negative.

These are just some possible explanations, and without more information, it’s difficult to know the exact reasons for the lack of communication.

8 Likes