We built a plugin that can fetch information for an entity from internet. So sometimes, a user asks “can you summarize A?” on ChatGPT. And we generated a very long JSON response to ChatGPT. What we notice is that ChatGPT can generate relevant and coherent texts for the first few sentences. And somewhere in the middle, it starts to hallucinate with completely unrelated and quite often wrong information. We hypothesized that it is due to the limitation of the context window?
I understand that we can always cut down the content we return to ChatGPT. But there are many cases where a long JSON response is needed (think a manual of doing something). Is there some way for us to still send the texts back to ChatGPT and help it avoid hallucination?
Is it possible for us to add prompts such as “read first 1000 words and summarize them into 200 words; and then read the next 1000 words and summarize them into 200 words; …”? Or something along this line?
Your help or experience sharing will be greatly appreciated. Thanks a lot!