Anyone noticing that ChatGPT will fabricate operationIds? It is not that it will use the wrong one, but rather it completely makes up operationIds. And sometimes there is a working valid openationId similar to the fabricated - i.e. real operationId might be getAllPostsById and ChatGPT will use getPosts. I am also finding that it is fabricating responses. For instance, Ill provide a return object with data ids and then it will use some of those for the next call and also make up some ids which then 404. Im not sure if it is related to the openapi.yaml file size, but I did notice these hallucinations once the file got around 1,000 lines or so.
Feels like how when you have had a long conversation with ChatGPT it will start to forget earlier parts of the conversation and then start making it up. Im guessing OpenAI is feeding the yaml file in as part of the conversation and aligning the system to pick the best api, but as the file grows it short circuits and hallucinates.
For what its worth - it consistently hallucinates. Funny enough, when I tell it “No, use this operationId” it will be like oh yeah, cool that works.
Hmmm… I do have descriptions, but will try to force the assistant. I thought I read in their plugin guidelines that we should not try to force chatgpt’s response like we do via the api. This feels sorta like that. Ill try it nonetheless and see how it improves
Ive tried the response forcing and improved my descriptions, but chatgpt still wants to use nonexistent functions. There is not one line of code that uses the function name it is attempting. Pure hallucination. Any other suggestions? It makes developing large applications for CGTP error prone and impossible to fix.