I’m encountering a recurring encoding issue using GPT‑4o via the Chat API (no streaming). The issue appears when the response contains non-ASCII characters like æ, ø, å. Instead of valid Unicode escape sequences (like \u00e6), I often get corrupted ones such as:
"Jeg har modtaget tilstr\u0000e6kkelig information til ..."
Expected:
"Jeg har modtaget tilstrækkelig information til ..."
This occurs intermittently, even with identical prompts and schema input. It appears before any post-processing on my side — directly in chatResponse.Choices[0].Message.Content.
Example from debugger:
ValueKind = String :
"{"JobAdCreateStatus":{"wascreated":true,"explanation":"Jeg har modtaget tilstr\u0000e6kkelig information til ..."
I’m using:
Model: gpt-4o
Endpoint: Chat completion
Tool use / function calling with JSON schema
SDK: Official .NET OpenAI SDK
Streaming: disabled
Encoding: UTF‑8 throughout
This behavior does not occur consistently, but often enough that it corrupts production content when characters like æ or ø are used.
Has anyone else seen similar corruption from GPT‑4o?
Any insights or official response would be very helpful
You’ll have to discuss the issue with the AI: that it should natively output unescaped Unicode UTF-8 language, even into structured output and code (providing the code encoding is Unicode, like Python 3), and also not high-ASCII code page bytes.
You can also put some example outputs in the system message, in the user’s language if known, or alternately some example chat turns (you can use "name":"example" on chat completions alongside role).
You can use logit_bias to demote certain tokens on Chat Completions, in this case, 7570 for “\u”.
Neither feature is provided with “Responses”.
You also have gpt-4o-2024-11-20 to try out.
This ultimately needs OpenAI to acknowledge and fix the models. One shouldn’t have to create an extensive fine-tuning to get the AI to properly write a world language.
Welcome new user with a web site to promote, multiple old thread bumper.
The AI can certainly write \u000000000000999 if it wants. Just as it could write the correct multi-byte Unicode code point if it wanted to and does in the majority of the cases for a topic creator that says “occasionally”.
The API cannot “ban escaped unicode sequences”.
The answer you seek: logprobs.
Get the actual predictions of tokens, and alternates that the AI is making.
You’ll see this is a training issue, likely large amounts of 8-bit code pages were ingested in corpus with flaky conversion on these models, two years old, and even more major fault with this symptom was seen in the initial release of gpt-4-turbo.
