Handling Overlapping Responses in Realtime API When Tools Take Too Long

Hi everyone!

I’m facing a specific issue while using the realtime API. When the model calls a function (tool) that takes several seconds to generate a result, the user might speak again in the meantime. This can lead to overlapping responses if the tool’s result arrives at the same time as the model’s new response.

Here’s a breakdown of the issue:

  1. t1 → The user asks a question.
  2. t2 → The model calls a tool.
  3. t3 → The tool starts executing.
  4. t4 → The user speaks again.
  5. t5 → The model generates a response.
  6. t6 → The tool’s result arrives, and the model tries to generate another response (which fails because it was already responding).

My Current Solution

I decided to ignore user input while a tool is executing, only processing it once the tool completes.

:white_check_mark: Pros:

  • Prevents overlapping responses.
  • Avoids mid-conversation interruptions due to delayed tool results.

:cross_mark: Cons:

  • This is not how a real conversation works. In a phone call, a user can ask something else while waiting.
  • It creates awkward silences and long wait times.
  • The user has no idea that my application isn’t listening while the tool is running.

A Partial Improvement

If a tool takes more than X seconds, I tell the model to generate a response like:
“I’m still retrieving the information, please wait a moment.”

Then, I re-trigger the same tool call with identical parameters. This way, when my app receives the re-triggered request, it knows a tool was already running, waits another X seconds (configurable timeout), and only generates the final response once the tool completes.

Key Considerations

  • I understand that tools should ideally be micro-tasks that execute quickly, but in reality, some computations or external API calls take time.
  • I tried adding a pre-call message (e.g., “Let me check that for you…”) in the tool descriptions, but it’s unreliable and doesn’t help for longer waits (e.g., 10+ seconds).

How Do You Handle This?

Has anyone found a better approach? Shouldn’t the model itself have a built-in way to handle this scenario?

Hi
I’m having the exact same problem.
I’d love it if you could share your experience with me on how you solved it in the end.
I can’t get it to say “Just a moment please” I always get an error of
{
“type”: “error”,
“event_id”: “event_Cwm3xsxJiXSNHmeUbIeIt”,
“error”: {
“type”: “invalid_request_error”,
“code”: “conversation_already_has_active_response”,
“message”: “Conversation already has an active response in progress: resp_Cwm3xbXuN6U8s9vn2KmXi. Wait until the response is finished before creating a new one.”,
“param”: null,
“event_id”: null
}
}
And I can’t separate the responses.

In summary, I implemented the following flow:

  1. Call the tool if it fires.

  2. Send a model preamble (e.g. “Let me check that for you”).

  3. Wait 3 seconds for the tool response.

  4. If the tool times out, return a tool response like this:

{
  "error": "TIMEOUT",
  "retries_remaining": 2,
  "next_action": "retry"
}

Then, send a prompt instructing the model to retry the same tool call with the same parameters (forcing the model to call the same tool again) and provide a preamble first.

  1. Set a second timeout of 4 seconds.

  2. Apply the same logic as in step 4.

  3. Set a final timeout of 5 seconds.

  4. Apply the same logic as in step 4, but since there are no retries left, return a tool response indicating the timeout error and send a prompt without tools, forcing the model to communicate the error to the user.

In this flow, if the user speaks while a tool execution is still in progress, I let the current execution finish. If the tool response arrives successfully, I append it to the conversation, and the model responds using both the tool response and the user’s latest message as context.

If the tool execution times out, I append a timeout error for that execution, and the model responds to the user’s latest message instead.

Thanks for coming back to share!

Thanks for sharing the workaround! Realtime supports continuing a conversation while tools run, but the response lifecycle still needs coordination. This guide covers response events and out-of-band responses: https://developers.openai.com/api/docs/guides/realtime-conversations. If the error continues, a redacted event trace would help Support narrow it down.