How can I pass additional context to the realtime API during a conversation

API
, ,
1

Is it possible to pass additional context to the realtime API during a conversation?

My app is a voice app (via phone/twilio), and I am using websocket connection to open AI realtime API.

I need to ‘tell’ the LLM when the call is nearing 5, 10 and 25 minutes so it can tell the user.

I tried using session update to pass this info as an instruction, but it doesn’t seem to work. The LLM doesn’t seem to get the instruction. Or at least it doesn’t act on it.

Any thoughts?

2 Likes
2

For getting it to tell the user something you should be able to send the response.create client event.

Very roughly, send a client event something like this (TypeScript):

const createResponseEvent: RealtimeClientEventResponseCreate = {
    type: "response.create",
    response: {
      modalities: ["audio", "text"],
      input: [
        {
          role: "system",
          type: "message",
          content: [
            {
              type: "input_text",
              text: "Tell the user that the call is nearing 5 minutes and ask if they would like to continue the call.",
            },
          ],
        }
      ],
    },
  }
  manager.sendClientEvent(createResponseEvent)
1 Like
3

Adding onto the answer of @activescott, do keep in mind that RealtimeAPI Sessions are limited to 15 minutes. For anything longer than that you would need to make a new session. :hugs:

4

Super helpful. thanks for this. Does this mean the call would have to be dropped and restarted?

1 Like
5

@activescott for the win!
Thanks for this. I am going to give it a try.

6

minor correction, session max duration is now 30 minutes
https://platform.openai.com/docs/guides/realtime-model-capabilities#session-lifecycle-events

1 Like
7

also try updating the instructions, I’ve found this works as a trigger in other scenarios

{
  type: "response.create",
  response: {
    modalities: [
      "audio",
      "text"
    ],
    instructions: "Tell the user that the call is nearing 5 minutes and ask if they would like to continue the call."

  }
}
1 Like
8

@damien.mcgivern Nice! I saw that there too, but hadn’t tried it yet. Anyone know if that works better or worse than a system message or is there any best practice or guidance on when to use which?

9

Exactly, you’d have to drop and start a new session (maybe with the system prompt being a summarisation of the previous conversation.)

Also, as @damien.mcgivern pointed out:

So you’d have to do it every 30 minutes. :hugs: