What does it mean that Batch API supports asynchronous requests?

API
1

In the docs(https://platform.openai.com/docs/guides/batch/overview)
the first line mentions “Batch API used to send asynchronous groups of requests”.

As per my understanding : Asynchronous requests are those that are sent without waiting for a response from the previous request. Once an asynchronous request is sent, the client can continue with other tasks without waiting for the server’s response. When the server is ready, it sends the response back to the user.

Users of batch api send the bulk of requests altogether which are executed by Openai (asynchronously I believe).
Then in Batch API , why do docs mention that users themselves can make asynchronous requests .

2

replace “client” with “user”.

3

Okay I did . Can you explain it now ?

4

Your question was "why do docs mention that users themselves can make asynchronous requests "

Assume you are a user. You submit daily batches to OpenAI. You don’t want to wait for each batch to be processed (i.e… you want to do other things while the batch is being processed). This is what it means for the users to able to submit async requests.

5

I see . Thanks .
Also can you tell me if is it correct to say that :
“APIs like of chat_completions and embeddings are designed for making synchronous requests in real time . There are no efficient ways of sending asynchronous requests through them .”

6

Looking at that statement from a purist standpoint, it follows the logical path. i.e. if they are designed for synchronous requests in real time, the designer is further making an asserting that they cannot be used for async requests efficiently.

Digging down further, the key issue is in the term efficiently. Take the example of my multi-agent framework. It certainly does makes calls to chat-completion framework asynchronously. But for that I have to have highly concurrent services that can trigger that asynchronousity. Further there is an invariable delay between someone making a request and that request getting acted on.

1 Like
7

So , can I say this :
"When the users have to make a bulk of requests , it is preferred to make them asynchronously through Batch API . "

And that batch API will be preferred when there are bulk requests only .

8

"When the users have to make a bulk of requests , it is preferred to make them asynchronously through Batch API . "

IF
one can afford to wait for a maximum of 24hrs AND
the batch requests can comport themselves to (a) completions (b) chat-completions (c) embeddings.