So with the new Batch API with the GPT4 vision model, how many images/text I can batch at most? and rate limits for this batch API?
The limit of batch tokens you can send depends on your API trust tier level (1-5).
To view the exact rate limits per model for your account, visit the limits section of your account settings.
The rate limit is given in maximum (daily) token queue. Queue basically means daily (unless you are programmatically monitoring the jobs) because they all will run at a preferred time of day.
gpt-4-turbo (with vision) has a significantly larger rate than gpt-4-vision-preview, which is not listed in the general tier guide.
Tier 1 (having paid OpenAI less than $50 in your account’s history) gives 900,000 tokens (of input) that can be sent to batch process. ($4.50 for just the input with the 50% discount)
The number of tokens consumed by the images range from 85 each (at detail:low) to 1445 tokens (at largest image and detail:high)
Then of course, your requests will have the messages that accompany the images, the task instructions the AI should perform with the images. Let’s conveniently say 115 tokens total for a role and simple instruction, for 200 tokens total per single low detail image request API call.
The maximum then for daily batching is 4500 API requests to be performed by the batch endpoint.
It would be prudent to break the job into smaller files, as base64 binaries of images may make for large files (you can resize first yourself), and you may want to ensure that the maximum number of jobs are processed, instead of all being refused for a single file mistakenly over the limit.