Realtime API - questions on client events and token usage: conversation.item.create, session.update, response.create

API
1

conversation.item.create

  1. what does it mean when a you create a conversation item with role=system? when would you want to create a conversation item with role=system?
  2. i’m guessing when you create a conversation item as user or assistant, then it’ll be like a manufactured conversation which didn’t actually happen, but for future user/assistant interaction, it will be as if it happened? am i correct to think that the longer the conversation goes on, the more tokens each successive message would eat up, whether from user or assistant?

response.create

  1. when should this event be used? i am currently using it for telling the user a specific phrase by setting the instructions field to something like “Tell the user: I am unable to provide the current weather in Kelvin”. can this event be used for something else?
  2. in the docs (https://platform.openai.com/docs/api-reference/realtime-client-events/response/create) the example they give includes a tool and tool_choice similar to the session.update event. when would I want to use a session.update to use a tool and when would I want to use a response.create to use a tool? If a tool is provided in the response.create but not in session.update, will the tool be available later or just for the latest response.create ?

session.update

  1. When would you want to send a session.update mid-session rather than just at the beginning? Does it make sense to send a new session.update to control the flow of the conversation? e.g. new set of instructions and new tools once the user has passed a certain “stage” in the conversation? e.g. if you’re simulating a job interview with a candidate where the realtime api serves as the interviewer and user as interviewee
  2. response.create and session.update both have an instructions field. How do these events differ? Do the instructions in response.create override the instructions in session.update?

token usage

  1. am i correct to think that generally, the longer the instructions in response.create/session.update or the longer the message length in conversation.item.create or the longer the message history or the more tools provided then the more tokens will be used up by each new interaction/event? are there exceptions to this?
  2. are there any ways to minimize token usage other than keeping instructions/message /message history as short as possible while still getting the job done?
3 Likes
2

I am similarly confused.

I’ve found that the realtime API is unreliable for data entry, so I’m using a transcription model + GPT-4 on the side to collect serial numbers.

I’m trying to “inject” these serial numbers back into the realtime context - but I’m unsure of what mechanism I should use to do so.

I’ve tried all 3 that @mcpower2 has listed above (conversation.item.create, response.create, and session.update), and response.create has been the most reliable at listening to the corrected serial numbers that I’m sending.

However, response.create only adds that information to memory for the next response, so the model forgets the serial number immediately after it spells it out to me.

What would be the recommended approach here?