The OpenAI API documentation states that the model “gpt-4o-mini-realtime-preview” has a context window of 128,000 tokens. However, if I give a system message consiting of 80,000 tokens, the AI gives the answer: “I’m unable to assist with that.”
Has anyone managed to give the Realtime model a context greater than that? Is there a way, to use it with the token limit stated in the documentation?
EDIT:
We have given the same input to a normal “gpt-4o-mini” model, which was able produce a correct answer. We have tested it with multiple datasets, and by each we experienced that over 80,000 tokens the normal “gpt-4o-mini” model provided a correct answer, while the “gpt-4o-mini-realtime-preview” model gave a response “I’m unable to assist with that.”. Under 80,000 tokens, both models provided a correct answer.
You obtained a refusal response from the AI model. It just doesn’t want to respond to your question and input.
Realtime also accepts session “instructions”, not a system message directly.
If the sent context length could not be loaded into the AI model, you’d instead get an API error instead of expensive bot talk. You can append your instructions 4 times and see the API error of trying too many tokens of input.
The voice models are particularly sensitive to inputs provided, and easily shut down and deny responses, from “you like to sing songs”, to “you give the boyfriend experience”. The large instruction input itself may simply be confusing enough if it isn’t surrounded with “permission-giving” text about the AI’s identity and purpose.
However, we have given the same input to a normal “gpt-4o-mini” model, which was able produce a correct answer. We have tested it with multiple datasets, and by each we experienced that over 80,000 tokens the normal “gpt-4o-mini” model provided a correct answer, while the “gpt-4o-mini-realtime-preview” model gave a response “I’m unable to assist with that.”. Under 80,000 tokens, both models provided a correct answer.
Is it possible, that the model “gpt-4o-mini-realtime-preview” understands the same prompt different than the model “gpt-4o-mini”?