We have a lot of different processes we run on old property documents, and all use pretty demanding prompts.
When processing these via API calls, the vast majority are successful, however, when processing the exact same requests in a batch, we get a failure rate ranging from 10% to 70%.
We have consistently experienced this across many different prompts and process types.
The requests used on both, the API calls and in batches, are EXACTLY the same and we’re using the ‘gpt-4o-2024-08-06’ model.
Can anyone please lend some insight on this and what a solution may be?
Thank you.