Assistants API too slow for realtime/production?

API
4

On my side, I’ve seen the same thing, but the output is taking ages to come.

My code looks like yours, even if it’s node.js

console.time('createAndRun')
const run = await openai.beta.threads.createAndRun({
  assistant_id: assistant.id,
  thread: {
    messages: [{ role: 'user', content: transcription.text }],
  },
})

// wait for the run to be completed via its status
let status = run.status
while (status !== 'completed') {
  const newrun = await openai.beta.threads.runs.retrieve(run.thread_id, run.id)
  status = newrun.status
}
console.timeEnd('createAndRun')

which gives me, for a small answer of two sentences : createAndRun: 17.337s

17 seconds seems to be really large.

I hope it will slow down

1 Like