Gpt-realtime-1.5: text output mode broken when tools are enabled

API Bugs
,
1

I’ve been using gpt-realtime-1.5 for a couple of days now and ran into an interesting issue. When using output_modalities=[“audio”] , the model works great. But when I switch to
output_modalities=[“text”] with tools enabled and rely on an external TTS, the performance drops significantly compared to gpt-realtime.

Issues I’m seeing in text-only mode:

  • Model wraps normal conversational responses in curly braces {} as if it’s outputting JSON
  • Function call arguments leak into the text output channel (the TTS literally tries to speak the function call JSON)
  • Internal control tokens leak into the output, e.g.: <|aesthetics_3|><|has_watermark|>
  • Ignores language instructions that gpt-realtime followed perfectly

None of these issues exist with gpt-realtime in the same configuration, or with gpt-realtime-1.5 in audio output mode. Seems specific to text mode + tools.

2

I would like to second that there is something very very wrong with output_modalities=[”text”] on the new model. Almost every response it gives is somehow wrong, or is a tool call at the incorrect time. After an incorrect tool call or response, it follows up with an “oops, I messed that up, let’s try again” and tries to continue.

3

Yes this is happening to me too , and for some reason the first turn/message is always stuck till a second message comes

4

Hi and welcome to the community!

I can also reproduce several of the behaviors you described:

  • In text-only mode, the model does return JSON-like content (for example, normal replies wrapped in { ... }) instead of a natural conversational answer.
  • I also see tool-related JSON leaking into the user-facing text output in this setup, which would cause an external TTS that reads the text stream to literally speak JSON.
  • In the same configuration, I see weaker adherence to instructions compared to audio output mode.

Will ping the team to take a look!

Ps. I did not capture the “internal control tokens” leak (<|aesthetics_3|><|has_watermark|>) in my tests. If anyone can share request IDs that will be helpful.

5

Thanks for reproducing and escalating!

Unfortunately I didn’t capture the specific request IDs for the control token leak at the time I’ll start logging them and share as soon as I can reproduce it again.

6

I can also confirm all these bugs happen in almost any conversation, regardless of the instructions the model gets. For me, as of now, gpt-realtime is superior.

Model wraps normal conversational responses in curly braces {} as if it’s outputting JSON

Function call arguments leak into the text output channel (the TTS literally tries to speak the function call JSON)

Internal control tokens leak into the output, e.g.: <|aesthetics_3|><|has_watermark|>

Ignores language instructions that gpt-realtime followed perfectly
7

Hey everyone, can someone please provide the request id so we can look at our backend and get this reviewed by our engineering team please!

8

I just saw this on session: sess_DDe8ALUUKkXHDI1OPQcmC . I don’t think I have a request ID, but that session is an example where it returns weird control tokens in the output, and not much else happens.

9

Hi Parashant — seeing the same issu and dont have a request ID either. But have a sessions id:sess_DGDRIlsIi4vdwRDiDmu2c

Would be much obliged if you had any feedback from the engineering team. Very eager to switch to 1.5 in production, but this is a blocker.

Thanks!

10

I’m seeing the exact same thing with out of bound transcriptions (ie. text output).

Reproduce by following the official realtime_out_of_band_transcription cookbook example and upgrade from gpt-realtime to gpt-realtime-1.5.

11

I’ve just captured the internal control token leak:

{
  "type": "response.done",
  "event_id": "event_DK5oBDolQHiBSwQL4ftMo",
  "response": {
    "object": "realtime.response",
    "id": "resp_DK5oAo6Y86oxkM497RSl0",
    "status": "completed",
    "status_details": null,
    "output": [
      {
        "id": "item_DK5oAevNakw7jAo1isZA9",
        "type": "message",
        "status": "completed",
        "role": "assistant",
        "content": [
          {
            "type": "output_text",
            "text": "<|aesthetics_5|><|has_watermark|>"
          }
        ]
      }
    ],
    "conversation_id": "conv_DK5nsjmPxL7hHjKzJQrmB",
    "output_modalities": [
      "text"
    ],
    "max_output_tokens": "inf",
    "audio": {
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "alloy"
      }
    },
    "usage": {
      "total_tokens": 5804,
      "input_tokens": 5772,
      "output_tokens": 32,
      "input_token_details": {
        "text_tokens": 5753,
        "audio_tokens": 19,
        "image_tokens": 0,
        "cached_tokens": 4864,
        "cached_tokens_details": {
          "text_tokens": 4864,
          "audio_tokens": 0,
          "image_tokens": 0
        }
      },
      "output_token_details": {
        "text_tokens": 32,
        "audio_tokens": 0
      }
    },
    "metadata": null
  }
}
12

Happening to me aswell. Would love for this to be fixed.

13

Hi,

I checked internally and our engineers have pushed a fix for this to production. Would someone be willing to try again and update me if your still seeing the same issues?

14

I just tested out-of-bound transcriptions are working with gpt-realtime-1.5

thank you!!

15

That's great to hear! With this in mind I'll close this thread

16

That's great to hear! With this in mind I'll close this thread

17

Hi,

I wanted to follow up as I’m still experiencing a related issue as of April 17th.

I am not using function calling, but I am using modalities:[‘text’]. The model wraps normal conversational responses in curly braces {} as if it’s outputting JSON.

This issue only occurs with gpt-realtime-1.5 and does not reproduce on gpt-realtime.

Sample session IDs for reference:
• sess_DVUR4gc9b8FQElVIMIlWG
• sess_DVCtlEJfVpYbAIEIHk7ZV

Could you please take another look? Happy to provide additional details if needed.

18

This original issue has indeed been fixed for me as well. However, I noticed a related issue that I started a new post for where the text responses include the word “analysis”. I thought I’d post here to see if anyone else in this topic has been seeing the “analysis” issue as well. - Gpt-realtime-1.5 text output_modality - drops words and replaces with "analysis"