When submitting chat completions requests for LLM-generated responses, we’ve found that having brief instructions that can be prepended to the last utterance submitted can greatly improve LLM response behavior. Relying solely on the system prompt at the beginning of a long conversation seems to indicate degradation in the generated response (as though the instructions are forgotten or have less influence). By adding important prompt directives in front of the last utterance in the messages array, we have a “current” reminder that the LLM seems to honor.
The Realtime API doesn’t provide a way to pass additional fields in the session.update message (though it doesn’t seem to be rejected). We have added an “utterance_prepend” text field so the client can override anything loaded on the server. On the server, we add this to the last utterance just before submitting it to the agent LLM, but we do not store it in the chat history we maintain. Because we used the session.update we only allow changes for the entire scope of the conversation as we weren’t sure if we could send an arbitrary session.update in the middle of a conversation. If this is possible, there are nice options to help guide/improve the conversation and generate responses.
Finally, I’ve hoped that the description of a tool would provide a prompt of sorts for the LLM’s tool-calling mechanisms. This doesn’t appear to be the case. For testing purposes, we have a simple tool that returns the time of day. We have tried to tell the LLM to always call the tool when time or data information is requested (so it won’t find the reference earlier in the chat and echo it back). Could we consider a formal way to have tool-specific prompts? I don’t want to have to add tool-specific details to the general system prompt, as tools will grow, and it seems their directives are only needed when the LLM decides whether to call a tool.
