My biggest pain point with GPT coding: outdated libraries, apis, docs

Prompting
1

My biggest pain point using GPT for coding is that it often uses out of date (and sometimes made up) parameters/endpoints for APIs and methods of libraries. It’s often hard to correct. What are the best techniques for getting it to stop doing this? Are there good techniques for feeding it documentation? Anybody found good articles or info about this?

1 Like
4

It seems it generally points to outdated datasets first but it has acces to new ones but you seem to have to force it to use it.

I compared a simple example:

OpenAi - Create a example tanstack table : uses deprecated v7 (usetable hook) ( like this thing is 3-4 years old
Claude - Always Uses V8 (usereactable hook)

I have to specifically ask for latest datase

5

Claude also has this problem, especially if it was not in the database. For example, I’ve forked some library, then I tell the model the changes and give an example. 50/50 change if it works in Claude. GPT-4o is slightly better when explicitly given instructions.

I think it reveals something fundamental about LLM’s. On traditional metrics they are probably heavily overfitted, but we don’t think that way since there is so much data available. I see same phenomenon when evaluating the ability to forecast, for any other algo, I’d say the results indicate overfitting.

6

Yes. It reveals that the are GPT, where the middle letter is “Pre-trained”.

The knowledge comes from a long, expensive training process. There is no database to simply add to. There are terabytes of training corpus and months and millions spent developing the model and performing post-training after that.

7

Kind of, but I don’t think it needs to be that way. Ada to Davinci didn’t feel that way. It is very useful that it “knows” things, but also a disadvantage as it can override information from the inputs per above conversation. Subjectively we see it in coding, but it probably also affect RAG based flows and potentially reduces finetuning performance. It also limits the usability of it for forecasting.

8

For some reason, Claude AI has all the updated API but not ChatGPT. I’m having to use Claude AI to help fix this.

9

It also helps to say which version you want, if you know.

For example, when asking for Python code using SQLAlchemy, I always specify SQLAlchemy 2, otherwise it often uses version 1.

It’s even enough to include “SA2” for the model to understand.