My biggest pain point using GPT for coding is that it often uses out of date (and sometimes made up) parameters/endpoints for APIs and methods of libraries. It’s often hard to correct. What are the best techniques for getting it to stop doing this? Are there good techniques for feeding it documentation? Anybody found good articles or info about this?
1 Like
It seems it generally points to outdated datasets first but it has acces to new ones but you seem to have to force it to use it.
I compared a simple example:
OpenAi - Create a example tanstack table : uses deprecated v7 (usetable hook) ( like this thing is 3-4 years old
Claude - Always Uses V8 (usereactable hook)
I have to specifically ask for latest datase
Claude also has this problem, especially if it was not in the database. For example, I’ve forked some library, then I tell the model the changes and give an example. 50/50 change if it works in Claude. GPT-4o is slightly better when explicitly given instructions.
I think it reveals something fundamental about LLM’s. On traditional metrics they are probably heavily overfitted, but we don’t think that way since there is so much data available. I see same phenomenon when evaluating the ability to forecast, for any other algo, I’d say the results indicate overfitting.
1 Like
Yes. It reveals that the are GPT, where the middle letter is “Pre-trained”.
The knowledge comes from a long, expensive training process. There is no database to simply add to. There are terabytes of training corpus and months and millions spent developing the model and performing post-training after that.
Kind of, but I don’t think it needs to be that way. Ada to Davinci didn’t feel that way. It is very useful that it “knows” things, but also a disadvantage as it can override information from the inputs per above conversation. Subjectively we see it in coding, but it probably also affect RAG based flows and potentially reduces finetuning performance. It also limits the usability of it for forecasting.
1 Like
For some reason, Claude AI has all the updated API but not ChatGPT. I’m having to use Claude AI to help fix this.
It also helps to say which version you want, if you know.
For example, when asking for Python code using SQLAlchemy, I always specify SQLAlchemy 2, otherwise it often uses version 1.
It’s even enough to include “SA2” for the model to understand.
1 Like
yep … and ai assisted coding becomes even worse. I mean you ask Chatgpt to produce a simple OpenAI centric chatbot for testing and you get deprecated openai calls … openai playground .
Sure you can fix, but it goes to illustrate the problem with never ending api changes … not just chatgpt … just about all.
1 Like
I have the exact same problem. It’s terrible for me. All LLMs I have tried cannot wrap their head around the inconsistencies of different versions.
Even when I say use the latest or a specific version, I still can’t get code that compiles even after giving the compiler error messages and trying the fixes.
I have to always fix things myself. And if I ask about something I don’t know then I am stuck because then I can’t fix it without first learning how.
It looks good on paper but just doesn’t work in the real world.
1 Like
I agree with you @torronen that “overfitting” is, more or less, the right word for this. When you give it fresh information about what to do, that differs from its training data, it should be able to overcome its training data and adjust. When it doesn’t, I think “overfitting” is a good term.