For the chatbot I am building using GPT 4o and 4o mini models; I provide different tool calls to the API, however sometimes (around 3/10 times); it replies with something like: “just a moment; let me do it for you. Doing it now … ; okay done.” which is a pure hallucination. If i look into the logs; its response doesn’t even include any tool call but it generates such messages out of thin air. I have set the temperature to 0.0 but the model is still performing unreliably.
What might be the best areas I can put my effort into to tackle this issue?
I got the same problem. Haven’t solved it yet.
I would advise strongly to have better tool description so the model knows exactly what it’s doing. Make sure to have the system prompt short concise and clearly defined. Try outlying the objective properly in the system prompt. Outline the system prompt in such a way that the tools and their description aids to solve the objective. Better prompting always helps. Temp close to 0 maybe like 0.01 and check for the results. Hope this helps 
1 Like