Unexpected reference tokens appearing in assistant output

[Lionnel]

We are using ChatKit with the OpenAI-hosted backend (Agent Builder workflows), with the agent model set to GPT-5.2.

In some assistant responses, we observe unexpected tokens (e.g. filecite, turn4fileX) that resemble internal reference or citation markers, as shown in the image below:

These tokens are rendered verbatim in the final assistant output and are visible to end users. The Show search sources option is unchecked in the agent settings in the Agent Builder.

Based on our observations, this issue occurs when the agent retrieves or uses data from files via the File Search tool. It does not happen every time and only occurs occasionally, but when it does, the tokens appear directly in normal assistant response.

This issue is also being noticed by our clients who are using the chatbot in production. When these tokens appear in responses, they are visible to end users and are perceived as a clear system error, which negatively affects the overall user experience of the chatbot.

Additionally, I tried switching the agent model to the newly released GPT-5.4 today. The issue is present there as well and, based on our initial observations, it even seems to occur more frequently.

[_j]

They are citation references. However they are being written wrong. Where the AI needed to have written a (recycle/refresh) unicode code point to finish the single citation, the AI instead produced a repeat of the (outline star) unicode code point.

@_j

The “ugly things” are the AI writing its internal citations container, and looping with more file chunk references (that is invalid) with the wrong Unicode instead of closing them with the correct UTF-8 character that is prompted. The same symptom has been observed before.

@_j

This is an unlikely prediction, as it is not going to be trained and will have little meaning. That is likely why the AI will not close a citation, but instead outputs the middle character again, then a pattern of writing another citation index.

@_j

They are meaningless Unicode private-use code points, and OpenAI is attempting to use them with a semantically-based language model. The AI apparently has to get really desperate to have nothing else to write before getting out of a container with the instructed U+E201 code point, instead of repeating what it just wrote.

You are a consumer of OpenAI’s poor vector store product, internal prompt language, and choice of characters the AI won’t produce reliably, and using AI that won’t believe that you have the rights to tell it how to use tools in a “developer” message to improve the situation.