AI Nostalgia: Remembering When Frontier Models Were *Really* Broken

I was thinking these days about how much we complain about the quality of frontier AI models.

Why am I using so many tokens?
I only asked it to analyze my small repo with 100k lines of code, and it cost me a fortune!
Outrageous!!

Then I remembered how, not that long ago, around the launch of the revolutionary “o3”, I was grumbling about how the model couldn’t return a complete piece of code.

I spent several threads asking:

“Please do xyz.”

And it would reply with something like:

“Oh, just change that on line 43 with this, then before function xyz add this line, and for the final touch, change line 84 with this…”

Then, after a long sigh, I’d reply:

“Okay, please give me the fully updated code.”

The code was barely 200 lines long.

And, of course, it would stop halfway through:

“An error occurred. Please try again later.”

Not to mention that we had something like an incredible 30 messages per week or so on the paid plans.


But as humans, we do love to complain, don’t we? :grinning_face_with_smiling_eyes:

There is something a little nostalgic about looking back at those early AI frustrations now.

Since we don’t have infinite tokens (yet), how about sharing some stories to help us appreciate how much things have improved since then—and laugh a bit about the tales we’ll tell future generations about our early adventures with broken AI?

Do you have any stories about early AI that you remember fondly?

This discussion was sparked by memories of “davinci”.

Remember when frontier models were good? Look at the davinci-002 excuse that replaced it - likely a drop from a 175b engine to 20b or less, making base models pointless.

Completions API Playground - share link


OpenAI models like o3 have their output length severely gimped, and then gpt-4.5 also was good but then kneecapped in output length - which likely would have continued if there wasn’t immediate competition from Gemini 2.5 Pro - a revelation in frictionless delivery of what you needed on-demand, that didn’t devolve into “# rest of your code here”.

DaVinci: Here’s the edited version with the corrected realloc().

Me: No! You didn’t allocate a new memory block.

DaVinci: I am sorry. Here’s the actually edited version with the corrected realloc() and the new memory block.

Me: Oh! I see it now. You actually did it and I missed it. Sorry!

DaVinci: Yes, I am sorry, too. I will get a cup of coffee because I am tired.


The foreshadowing undertones of this brief conversation haunt me until today.

Back when Brockman and others answered supports emails! :wink:

Not sharing the reply email, but I got early access to the GPT-3 API! :wink:

Way back in GPT-2 I was playing around with “structured data”… the [[TAGS]] didn’t show up in training data very often, so they worked well for getting (somewhat) structured data back from the Davinci completion models with a one-shot or two-shot example. Other times it would invent its own tags.

Those were wild days… How far we’ve come!

One of the first things I did, back in February 2023, was spend a few weeks exploring ChatGPT’s capabilities and limitations with the programming language SWI-Prolog.

That work is still available as a set of Discourse Wiki pages, starting here:

Great initiative @aprendendo.next :raising_hands:

One of the funniest memories was when I worked with personas with gpt-4o using Finnish and it could be quite catastrophic. It made so many strange output mistakes and could hallucinate with complete confidence, such as files that I attached and ask it to summarize them. Yep…thank god I am not trusting and even more grateful that today’s models are more reliable and a lot better in Finnish.

The absolute funniest was when it once referred to a hippo as “läskipossu,” which basically means “fat pig” in Finnish even though my input specifically said hippo…don’t know what happened there​:sweat_smile:

But strangely enough, it was really good at holding personas consistently.