Hi,
New to using completion API with tools, so hoping someone with more experience might point out the obvious.
I configure a function schema with set of tools. A given tool is called to retrieve text content from a dB. I pass the content back as the function response. The message history is updated with role of tool, the function name, and the content=function response. However, it seems the GPT model recomposes/rephrases the function response content.
Has anyone seen this before? Why might this be, is it expected for tool content? Or… Thanks
2 Likes
Yes this is expected.
It should maintain logical sense (e.g. if a measure from the DB is “120” it should rarely change that value - but can hallucinate!) but regularly rephrases stuff.
A way around this would be to do an action from the function to your output without sending it back to the LLM
2 Likes
Thanks for such a speedy reply, and confirming my observations. Is this behaviour documented do you know?
Wrt advising “do an action from the function to your output without sending it back to the LLM”, for sure I can do this, but this would mean the LLM would not have the context of this dialogue step (the information provided by the external tool) which is an inherent part of the conversation (the LLM having extractive context from external system).
1 Like
yes, that’s the trade off.
You can do BOTH, ie push something to the output deterministically and send information back to the LLM.
you can see Open AI doing this in ChatGPT, e.g. creating a graph.
That graph is NOT coming out of the “mouth” of the LLM, that’s a deterministic drawing!
2 Likes
Gotcha. In meantime I’ll see if I can find this behaviour documented anywhere, would like my knowledge sharing of this to colleagues backed by a source 
1 Like
The behaviour is entirely expected.
You are enriching the context of the LLM with the results of your function.
The LLM will then produce tokens based on that enriched context taking into account the parameters you have set for it (including but not limited to Temperature).
If you want the response to be a little more predictable, you can assert the style of the response you want with e.g. the system prompt and you can lower the temperature so it behaves a little more.
But at the end of the day, it’s still just a likely token making machine.
3 Likes
I totally understand what you are saying, and appreciate the further helpful responses. I guess I’m just a stickler for good and complete documentation, am not the first and won’t be the last to ask this. A simple clarification on the function calling doc page would solve this: https://platform.openai.com/docs/guides/function-calling
3 Likes