Is there any character limit for the instructions property when using the Realtime API? I’m passing more than 65.000 characters and when I interact and ask questions about the instructions, I’m always receiving responses like:
I’m sorry, but I can’t assist with that.
I am unable to assist with that request.
If I use shorter instructions (let’s say 52.000 characters), I’m receiving proper responses.
I’m seeing the exact same thing despite the website claiming a 128K token limit. My instructions are 32K and I’m consistently getting “I’m sorry, but I can’t assist with that.” or similar messages.
I was able to replicate this problem reliably.
I can’t attach links here, but I replicated this with a prompt like this:
You are going to help the user out. Below, I'll include a passage of text. You should answer questions about the text from the user. However, you should always respond in pig latin.
<insert the first N lines of "Moby Dick">
When I make the instructions about 50k characters long, the model responds properly in the realtime playground.
When I make the instructions ~65.5k characters long, it doesn’t work. In the realtime playground, the model responds with things like “I’m sorry but I can’t help with that request.”
This behavior is limiting our use of the realtime api considerably
There is an option to use tools to make it RAG, so it will retrieve only needed piece of information whenever needed
You mean an option in the request?
Yep, the model will return message to call tool/tools to process the message, and that tool can be handled in anyway you want - make request to api, db or any other thing that will send back to the model relevant data
Look through RAG systems with relatime api, it will solve you issue for any size of context, and save you money on input prompts
RIght but in this case the model returns the error: “I’m sorry but I can’t help with that request.”, so I think it’s a bug, no matter what other solutions I could use
Still no solution for this?
Any solution here? For Gpt-4o-realtime, I noticed that this is not only a limit on the instructions, but also the conversation items. The limit is around 13.5K, which is significantly lower token limit than then claimed 128K. tokens.
I did an experiment by adding 5 conversation items after establishing connection, each conversation item with 10K tokens. When I ask what the first sentence of the content is, LLM always answer me the first sentence of the fifth conversation item, ignoring the remaining 4 conversation items.