I’m calling an inference API provided by a third party, for the most part it seems like a 1:1 copy of the completions API minus logprobs. (It just sets them all to null) I have a json_schema response_format that has 3 fields, among them a “message”, but I’ve noticed anytime it’s about to produce a …

You can try re-sorting the fields in your JSON schema. See if the AI is more likely to continue past a colon in a later key’s value. If you aren’t paying for unseen tokens (send the entirety of the output string through a tokenizer ) It does sound like something about the model weights or the strict…

The first thing I’d do: see the output token count that is being returned, compare to the text that you get run through the o200k-harmony tokenizer (likely same as o200k_base). Is there much more to be seen as output than you see? Then log the wire transfer outside of a calling SDK, see the raw co…

So with the response_format I’m getting around 170 tokens, without it 489 (valid response). Seems to indicate it is messing up somehow? I’m already making the API request myself. I put the ellipsis there to replace other valid JSON, they do continue, in theory they can be produced before the messag…

Unseen: the model is generating, such as writing the full string of the JSON key, but for some reason, you never get the output transmitted to you.

Well the reordering of the response_format keys worked, so I’m even more baffled now, but I’m just going to take it as a win, and contact the provider about this incase they’ve any insights, thank you so much for your help!

Third Party API Truncates Field Content Inside Response Format

API

zhayat October 22, 2025, 12:51pm 5

Sorry what do you mean by unseen tokens?

Topic		Replies	Views
Structured Output Issue in GPT-4o API – Response Truncation at Specific Index API api , structured-output	1	468	March 19, 2025
Response_format=json_object returns invalid json with finish_reason=stop Bugs json-mode	7	1031	January 7, 2025
Web Search Completion Cuts Off Response and ignores structured outputs on complex prompts API api , structured-output	7	1223	June 21, 2025
Invalid JSON response ending with API json	6	2117	March 8, 2024
Fine-tune 4o model - endless inference for JSON Bugs	6	309	April 18, 2025

Third Party API Truncates Field Content Inside Response Format

Related topics