Assistant API usage pattern

I am building an assistant for Bndtools/Eclipse. This is very chat like but the function calls can access the projects and files.

The strategy I found is this:

  • Poll the Thread for new messages, each new message found
    is added the window
  • Poll the Run for the completion status and function calls
  • Poll the RunStep for the progress

It seems kind of inefficient to have 3 polling process. I tried to use just the RunStep because that is kind of an event bus. However it lacks the information
about function calls. Theoretically it could be used to fetch the message but
unfortunately I get empty messages with an ID and then a little bit later the real message. Also, because the Run objects contains the instructions, it is unnecessarily large to send back and forth. The RunStep is much smaller.

And I get lots of 504’s right now.

How do other people architect this issue?

I don’t use runsteps at all. And only poll the thread when the run status changes. SO the only polling I am doing is for a change in Run status. Based on that either a function call is triggered, or a completion (and of course, you keep polling, until completion, or another function call)

But how then do you know there is a new message? This is only in RunStep?

And that means you only get the messages at the end of the run?

Mmm the status changes to either ‘completed’ - in which case there is a message. Or function calling needed
So until the status is completed - there will not be a message.

Right now there is a partial outage that may be affecting results, see status.openai.com for details.

No that is not true.

During a run, the assistant can add messages. For example if a function call fails or it takes another approach, it will post a new message. These new messages are announced in the RunStep. The RunStep also tells me if the retrieval or search function is used. Since a Run can take some time, this is a nice way to feedback to the user it is still running ok.

In my code, whenever the chat is started, I poll the messages and just add each message to the window. This works very well. Even the user messages go via this path. I.e. I do not add them myself to the chat window. They are submitted on the thread and then will show up.

I would like to just use RunStep … during the run. It could signal new messages and that there is a function call pending. Since the messages are small, I could poll at a high rate and only use the larger messages when it is really needed.

1 Like

At some point, streaming is what will ‘solve this’, once it makes it to Assistants.
I run 95% of my Assistants in non real-time environments (does ‘doing stuff’ in the background, reading / answering emails, updating databases).

I run 95% of my Assistants in non real-time environments

Ah, that makes a difference. I’ve got an anxious user idling :slight_smile:

With the quality of the response I get from gpt sometimes, I guess if I’d be the developer of your model I’d be the one that would be anxious :slight_smile:

Thanks.

It’s a matter of ever more detailed prompts. I have learned the last few months that humans are actually pretty bad at giving good instructions. :slight_smile:

1 Like

It’s a matter of ever more detailed prompts

Yeah, prompting is still black magic to me.

I notice that with more exact formal instructions it actually gets worse. So far, trying to stay in the area of common knowledge in the LLM works best. For example, I had an API with access to files and making it very unix command like seemed to work better than have a very different style.