I’m using the OpenAI Batch API and noticed that one of the requests in my batch failed with the following error:
504: Timed out generating response. Please try again with a shorter prompt or with `max_tokens` set to a lower value.
This request does not appear in the output file because it failed.
My question is: Is this failed request still billed? Specifically:
- Are prompt tokens charged even though no output is returned?
- Are completion tokens charged if the generation timed out?
I’d appreciate any clarification on how billing works in this scenario. Thanks!
Hi btvinh, that can definitely be confusing.
For Batch API requests, billing is based on token usage from completed requests.
If a request fails and only appears in the error file without a response or usage data, it isn't treated as a completed request, so prompt tokens shouldn't be billed just because the request was submitted.
Similarly, if a request times out, completion tokens are only billed if they were actually generated and recorded in the usage data. If the request fails with no output and no usage object, completion tokens shouldn't be charged.
The clearest guidance is that, for expired batches, you're only charged for tokens consumed by completed requests. Failed requests are returned separately in the error file.
Checkout this documentation for more info.
Avinash