Batch API is now available

Announcements
21

I am a bit confused why the Unix timestamp fields (created_at,in_progress_at,…) in the spec are string: update spec for batchapi by hihihuhu · Pull Request #226 · openai/openai-openapi · GitHub when the examples are an integer?

22

I am wondering what types of tasks folks are thinking about with this batch API with a 24h return time.

I am thinking, is it processing massive amounts of data for:

  1. Classification/Sentiment
  2. Generating data to train or fine-tune another model
  3. Summarizing data
  4. ???
2 Likes
23

Translation might be another option.

image

2 Likes
24

For me it’s a great way to clean up & possibly modify semantically rich datasets once every few days or so. I embed each row for further analysis. I don’t mind at all having a <=24h wait period before embedding.

2 Likes
25

I think one nice option is to replace an immediate GPT 3.5 result with a GPT 4 Turbo result later on as a batch job … (if that’s beneficial and the specific task allows).

Then you have the immediate result, and the (potentially) refined result later on …

2 Likes
26

Hello I’m trying to use Batch API and I get this error returned

 {
        "code": "missing_required_parameter",
        "message": "Missing required parameter: 'custom_id'.",
        "param": "custom_id",
        "line": 18
      },

I uploaded a jsonl file which was formatted like this

jobs = [
    {
        "model": "gpt-3.5-turbo-0125",
        "messages": [
            {
                "role": "system",
                "content": system_prompt,
            },
            {"role": "user", "content": all_prompts[x]},
        ],
        "frequency_penalty": 1.1,
        "max_tokens": 4000,
        "temperature": np.random.choice([0.9, 0.95, 0.85, 0.87, 0.91, 0.92, 0.97]),
    }
    for x in range(n_requests)
]

If there’s a sample for chat completion task for BatchAPI that would be wonderful. Thank you.

27

The request input object example (aka upload file):

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is 2+2?"}]}}

JSONL means JSON separated by lines. All on the same line.

You must generate your own custom_id, so that you can match responses back up with the originating message (there likely is no guarantee that all calls will be in the same order or will be successful)

2 Likes
28

Does this batch API mean that it will produce better output than traditional processing? For instance, if I upload some programming code file, will it process continuously for 24 hours/less? Or is it just an async processing?

29

each line in the .jsonl must be as follows

{“custom_id”: “request-1”, “method”: “POST”, “url”: “/v1/chat/completions”, “body”: {“model”: “gpt-3.5-turbo”, “messages”: [{“role”: “system”, “content”: “You’re a helpful assistant.”}, {“role”: “user”, “content”: “What’s the capital of France?”}]}}

30

Hey _j,

Apologies for the delay! I wanted to make sure I had the correct answer for you given some of the recent improvements we were rolling out.

The file limit in most areas (including the Batch API) is 512 MB. Because the configuration of the Assistants API is slightly different, the file size limit in Assistants is somewhat unique in that it was either 512 MB or the 2M language token limit you mentioned. With today’s announcement and rollout of v2, that limit has now been bumped up. However that change is specific to the Assistants API, meaning the Batch API still adheres to the 512 MB limit.

2 Likes
31

A post was split to a new topic: How to Resolve Error Code 500 in BatchAPI Requests

32

For me the first priority would be a completion window that falls within a typical workday, say 6h. 24h is too long.

33

@adjokic Thanks for your response and verification of upload size.

At 512M, it will be possible for most to upload a $100+ of tokens file that first simply can’t be enqueued.

I tag because there’s an unclear point:

  • Do “list batch” reports expire or any plan to remove them? Any rush to get your file report with file id?

Navigation by pagination that can only be done by ID may become unweildy (like fine-tuning) if the list just grows forever without expiry or a delete batch report(s) endpoint for self-management. You’d have to remember the last position.

34

Hi, batch processing is an incredible feature that we’ve been waiting for at our company.
We’ve been experimenting ever since to find out a few things about the TPD limit. Unfortunately the FAQ is not that clear about it, or I didn’t find it.

We have around 30bn tokens for translating text into different languages. We want this task to be done within 2-3 weeks, which wouldn’t be possible, if we only have 300 Mio TPD (Tier 5).

Now my question:

  1. To what the TPD is bound exactly? Org, Proj, Key, User?
  2. Can we somehow request a higher TPD for a certain period of time?
  3. Is there another solution than just setting up new OpenAI accounts with new credit cards to overcome this?
  4. How long would it take, to become Tier 5 after setting up a new account?
  5. What is the Batch Queue Limit 300 Mio Token TPD exactly? Can you give an example?
1 Like
35

It gives error when I put this property. It seems that property not yet ready.

1 Like
36

The batch queue limit is for queueing.

The unanswered question is how often they will run the jobs and reduce your queue, which is unanswered except that the 24hr window likely means they are targeting their own 24hr low usage period to run these.

You can closely poll and watch the batches, and submit a new one as soon as one of equal size is completed. – However, the “per day” that is in the rate limit of your account likely means it is measured as the maximum input per day, and perhaps not the maximum queue depth at any particular time.

If you find that they only run at 2am, then there’s not much alternative to queue depth thus equaling your daily rate, unless you have another paid account that has already aged past the point where you can fund it to bring it to tier 5 also.

“Request an Exception” doesn’t have a batch option; you’d have to explain after picking TPN that you want the batch rate lifted so you can spend your minimum $7500+ with 50% off.

If you have a monthly spend that is more than doubling the monthly $10k, that’s where you can probably get a response from sales.

1 Like
37

Thanks @jeffsharris for the note.

I am wondering whether it is possible to delete batches. Otherwise, the list gets longer and longer. I understand that client.batches.list() is paginated. But the list can get really long.

Another question that I have is regarding the file(s) you pass to the batch. Can you delete the uploaded file(s) immediately after running the batch? Or the uploaded files are required until the completion of the batch?

Thanks.

5 Likes
38

Pretty interesting!

During the weekend i’ve done some experiments with 10 lines and it worked pretty quickly (probably 2-3 minutes using GPT3.5).

Using ChatGPT i’ve created some bash Scripts using Curl to quickly test the system.
What i’ve learned :

  • the upload must be done by code, because you need to specify the purpouse “batch” which is not supported (at the moment) using the playground interface.
  • You need to be very careful with the jsonl format, it’s easy to make a mistake: my advice is to use ChatGPT as a linter.
  • you need to read the result of the batch, parse the result file id and finally request to download the contents of the result file id… which is a jsonl itself.

This morning i’ve started a 4 lines esperiment…

     "created_at": 1713775850,
      "in_progress_at": 1713775852,

(which was started at 8.50am UTC, now it’s 13.42 UTC).

Just to be clear the prompts are really simple: “what was the color of the white horse of Napoleon”.

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4-turbo", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Di che colore era il cavallo bianco di napoleone"}]}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4-turbo", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Di che colore era il cavallo bianco di napoleone"}]}}
{"custom_id": "request-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4-turbo", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Di che colore era il cavallo bianco di napoleone"}]}}
{"custom_id": "request-4", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4-turbo", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Di che colore era il cavallo bianco di napoleone"}]}}

I’ll post an update when it finishes (hopely before tomorrow morning).

Would it help the community if i share the bash files ? (without api keys of course)

39

I think it is very much time dependent. Last night I had the same delays are you indicated . Today I have 6 batches in the queue and they are taking more than 4hr (still pending). @_j and @merefield also mentioned this… I think OpenAI has a distributed server setup. So, potentially late evenings and night are the best time for debugging purpose with lower latencies.

1 Like
40

Great, Thank you. I’m excited about the 50% <3