A theory on why GPT 3.5 & 4 have gotten worse

The current situation appears to stem from a combination of influential factors. Firstly, the European Union’s implementation of stringent regulations on data utilized for Artificial Intelligence (AI) is a significant contributor. This development is likely prompting a flurry of activity within the OpenAI team, as they scramble to sanitize and alter segments of their dataset to comply with these new rules. Furthermore, these regulations necessitate a complete overhaul of their data collection methods.

Secondly, the conventional wisdom that AI datasets require an increase in parameters to enhance performance is being challenged. This is evident in Google’s PALM 2, which has exhibited improvements over its predecessor, PALM 1, despite utilizing less data. Sam Altman, has echoed this sentiment in the past. The compounded pressure from the EU’s data crunch could potentially create difficulties for OpenAI in enhancing their dataset to the extent that Google has accomplished from PALM 1 to PALM 2.

Lastly, there is an observable shift in focus towards reducing the operational cost per token for GPT-4. As it stands, the cost of running GPT-4 is significantly higher in comparison to its counterparts. However, as AI technology continues to progress, it’s becoming increasingly possible to improve performance with smaller datasets and lower running costs. This trend may be driving OpenAI’s current strategy and focus.

1 Like

Indeed, cost per token is falling (everywhere) and as Musk suggests, it will eventually cost more to account for the token than the token itself, making inferencing near-zero cost. But this is not lastly; open-source is likely a growing factor swaying OpenAI’s strategy.