Is TPD Really Tokens per Day or Tokens per Batch

API
1

I’m working on batch processing with AI APIs and need clarification on how TPD (Tokens Per Day) limits actually work.

My specific question: If an API has a TPD limit of 90K tokens, does this mean:

  1. The maximum size of any single batch request is 90K tokens, OR
  2. The total cumulative number of tokens I can process across ALL batch requests in a 24-hour period is 90K?

For example: If I have 120K tokens I need to process, could I split this into two separate batch requests of 60K tokens each and submit them on the same day without hitting the TPD limit? Or would I exceed my daily allowance since the total is greater than 90K?

I’ve received conflicting information and would appreciate hearing from anyone with practical experience with these API limits. Thanks!

1 Like
2

The batch processing limit is the total outstanding queued and uncompleted requests.

The “TPD” added to the figure in the limits page is indeed confusing.

If you were to have 75k token batches in a batch job file, and wait until each is finished and completed by the API (within a few hours, perhaps) before taking any other action, you can keep on submitting more once the first is done. If you were to send another 75k immediately while the first is still queued and waiting, you’d get an API error.

Enqueued prompt tokens per model: Each model has a maximum number of enqueued prompt tokens allowed for batch processing.