My json response is under 38k characters and the response is hallucinating

I believe the docs call out “We also impose a 100,000 character maximum for the API response body which may also change over time.”

Basically Chat GPT makes up the teams and players and only partially uses the returned json (in no particular order). Has anyone else seen similar behavior.



It really depends on how you prompt if you get hallucinations or not, I’m coming back to your specific plugin problem later.

Ask ChatGPT for a book that doesn’t exist and insist that it exists and it should know it.
Same happens if the model doesn’t have knowledge about things that don’t exist.
I mean it is all about propability.
When you train a model on 20 sentences in which the word rides is followed by horse you have a high chance that after rides comes a horse.
If you ask the model then that there is a cow (and it doesn’t know what a cow is) and you ask the model e.g.
There was a cow named lisa and every time tom rides…
Complete the sentence

You should expect to get horses.

So that applies to the model in general.

Adding the information that a cow is rideable to the prompt doesn’t neccessarily change this behavior.

And I guess this is what is happening here. The plugin’s answer is added to the prompt.

When GPT-3.5-turbo came out I have tried something like taking the userprompt, summarize it, extract possible wikipedia entry title (like hey assistant please give me a list of possible titles), then search wikipedia for it and crawl the article, summarize/extract important data and give the summarized data together with the prompt in relation to the previous userprompt.

But it did not use the data given to it in every case. I guess that’s the same here.
Especially names, dates and stuff like that has been removed when summarizing the wikipedia entries.

Maybe you can shorten the answer and don’t overload it with informations?
Maybe add a new endpoint for details so chatgpt can first get a list and then some details?

Put the entire json string into the tokenizer here and see if it breaks the token limits OpenAI API