Hi all,
I’m facing an issue with batch processing using GPT-4o-mini.
The task I’m working on is split into pages, and I process these pages in batches. Each batch follows the same logic and setup, and all the pages in a single run are executed on the same system_fingerprint configuration.
However, I’m encountering a problem where some batches fail with the error:
"You are not allowed to request logprobs from this engine."
What’s strange is the inconsistency. Sometimes, a batch fails on one run, but if I retry, it might succeed. Other times, everything works fine for an entire task, but a few pages might fail with the same error on a different run.
Everything works perfectly when I use the same batch setup with GPT-4o or the chat.completions. This issue seems specific to GPT-4o-mini and happens (with logprobs=True).
Here’s an outline of the batch setup I’m using:
batch_information = [
{
"custom_id": str(request_id),
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": self.model,
"messages": messages,
"response_format": self.json_schema,
"logprobs": self.logprobs,
"top_logprobs": self.top_logprobs,
"temperature": self.temperature,
"top_p": self.top_p,
},
}
]
client.batches.create(
input_file_id=batch_input_file_id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
The configuration is consistent across all batches. I’ve double-checked the input, and I can confirm there’s no difference between the failing and succeeding runs in terms of how the batches are created or processed.
I have not found information on that issue with GPT-4o-mini or if there’s something specific about logprobs with this model.
Would like to hear any thoughts or advice. Thanks in advance!