Issue with GPT-4o-2024-08-06: Repeating Tokens in Structured Output

API
2

Hi!

If you want to hope to talk to someone for a refund, you’ll have to click support top right here and persevere through the bots

A section of a webpage header shows navigation links for "Docs," "API," and "Support," with a title below reading "04-08-06: Repeating Tokens in Struc." (Captioned by AI)
A section of a webpage header shows navigation links for "Docs," "API," and "Support," with a title below reading "04-08-06: Repeating Tokens in Struc." (Captioned by AI)700×202 8.22 KB

This is not an uncommon issue with JSON/structured mode

This is a typical issue and often associated with a bad prompting strategy.

I would say that it manifests if the model is trapped in a specific state (in terms of the JSON finite state machine) but really wants to do something else but isn’t allowed to escape.

Here’s some things you can check:

  1. is your schema obvious and retrievable? If your schema is not obvious, or you have multiple schemas, or conflicting schemas, the models might struggle.
  2. is your schema straight-forward? The models can handle some complexity, but a simple, flat schema is typically your best bet.
  3. does your schema actually reflect the workflow? If your workflow asks the model to do something one way, but the schema forces it to approach the problem in another way, you might run into issues.

Here’s what I would do:

  1. Disable JSON mode/structured outputs
  2. work on your prompt until the model reliably returns the schema you expect
  3. optionally add logit biases: negative for ["```"], positive for ["{","["], depending on your model
  4. consider leaving JSON mode/structured output off - if schema validation fails, that would actually be desirable, because it implies that the model failed to understand or follow the schema

If you can achieve stability with this approach, you shouldn’t run into issues when you turn the restriction mode on.

TIP :bulb::
Consider adding something like “Your reply must be valid JSON, otherwise the system will break. Begin your response with {” to the very end of your prompt, the last thing the model will see before beginning its reply.

Hope this helps, good luck!

FUN FACT :face_with_monocle::
\u007f seems to be the unicode control character for DELETE (Unicode Character 'DELETE' (U+007F)) - It looks like you managed to get the model into a state where it tries to backtrack on its progress. Very cool emergent behavior!

2 Likes