Structured output on batch-api giving incomplete results

I am using structured output with BatchAPI because I have 1500 documents and it is cost efficient. 25% of my responses have incomplete jsons as output. For my response_format I used this function:

from openai.lib._parsing._completions import type_to_response_format_param

and gave it my Pydantic schema.

75% of the responses are fine and have proper jsons are results but 25% of the responses just cut the json off suddenly. This does not happen when I use normal API.

What could be the issue? There is no error on the response. Could the token limits have something to do with it? Im on Tier1 I believe and my prompts have the entire input document and a somewhat large pydantic schema. But again, this doesn’t happen in normal API usage.

ListofCategoriesofPICollected:[
{
text_span:
The type and manner of data that we and our vendors, including third parties like Google, collect includes the following: Personal Identifiers — When you sign up for our services such as name, company name, address, phone number, email address, billing information (name and address), number of employees or users and name of employees or users within your organization using our services. Please note that iWave does not collect credit card or other financial information — payment transactions are processed directly through Invoiced.com Internet and Network Activity — We and our vendors, including service providers and third parties, collect information about how you use our services and the computers or other devices, such as mobile phones or tablets, you use to access our services. Examples include: IP address, Unique device identifiers and device attributes, like operating system and browser type, Usage data, such as: web log data, referring and exit pages and URLs, platform type, number of clicks, domain names, landing pages, pages and content viewed and the order of those pages, the amount of time spent on particular pages'

Have you given the API call within a batch a “max_tokens” or “max_completion_tokens” parameter?

That parameter will cut off the output right at the length and cost you specify.

Inspect your batch results file itself, take the truncated content response, and paste it into a tokenizer. See how many tokens were sent.

Also check finish reason in the API response object, “length” means it hit your setting.

3 Likes

Thanks for the finish reason suggestion.

I noticed, the finished_reason for failed prompts is “content_filter” and when its successful its “stop”

1 Like