Malformed Message object points to issue with Assistants API

API
1

Hoping someone can help out here. I’m occasionally running into errors calling the Assistants API, where a conversation will just crash for a customer. I finally was able to track down a specific Thread where the issue occurred:

    {
      "id": "msg_MTh2gEfVqhxLXXXXXXXXXX",
      "object": "thread.message",
      "created_at": 1731602252,
      "assistant_id": null,
      "thread_id": "thread_59kyBrffVIH6kXXXXXXXXX",
      "run_id": null,
      "role": "user",
      "content": [
:
 Character and often focuses on their evolving relationship.\n\nLet's analyze these Throughlines based on the information found in the document. I will need to search for specific details to accurately frame each. Let's start with a search to identify these components from the document.",
            "annotations": []
          }
        }
      ],
      "attachments": [],
      "metadata": {}
    }

which comes right after the legitimate user response that I added and ran:

    {
      "id": "msg_MTh2gEfVqhxLdNXXXXXXXXX",
      "object": "thread.message",
      "created_at": 1731602252,
      "assistant_id": null,
      "thread_id": "thread_59kyBrffVIH6knXXXXXXX",
      "run_id": null,
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": {
            "value": "Ok. What are its Throughlines?",
            "annotations": []
          }
        }
      ],
      "attachments": [],
      "metadata": {}
    }

And right before a working response:

 {
      "id": "msg_8aFPZDPmwmIXXXXXXXX",
      "object": "thread.message",
      "created_at": 1731602254,
      "assistant_id": "asst_OEtqECBTu6XXXXXXXXX",
      "thread_id": "thread_59kyBrffVIH6XXXXXXXXX",
      "run_id": "run_sPQROY2EYX0gSXXXXXXXXX",
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": {
            "value": "To identify the Throughlines of the story, I'll break down the story into its main components using the Subtxt Narrative Framework.[...]This Throughline highlights the dynamic between the Main Character and the Obstacle
 Character and often focuses on their evolving relationship.\n\nLet's analyze these Throughlines based on the information found in the document. I will need to search for specific details to accurately frame each. Let's start with a search to identify these components from the document.",
            "annotations": []
          }
        }
      ],
      "attachments": [],
      "metadata": {}
    }

But check it out - do you see that line break after the word Obstacle and the next word Character? There’s some weird invisible character-thing going on there - because everything that appeared after - is exactly what showed up in that previous message attributed to the user role!

(Note: the […] was something I added to edit it here for the forum to make it easier to read)

The user never received this complete response - the streaming just error’d out and my app took care of it.

It’s very strange because that malformed one, which doesn’t have the typical type/text key/value pairs, actually shares the same message id as the id claimed by the previous user submission.

If it helps at all - my current approach is to add the user’s message first with this:

https://api.openai.com/v1/threads/{$this->thread_id}/messages

and then follow-up with this to stream:

https://api.openai.com/v1/threads/{thread_id}/runs

Thank you in advance for tracking this down!

2 Likes
3

I appreciate the help, but how could I possibly add the AI-generated response from my end as ‘user’, before I even received it?

6

I suspect that the AI has emitted a special token - and not a stop token, but perhaps one of the other high-number tokens that are used for the ChatML container format. The colon insertion can be the place for ‘name’ field that appears after a role, and the fact that it continues with capitalization means the AI has recognized the quirk of completing right after a special token or a newline, not putting a normal word token with a leading space there.

The lead-up language is also odd; the AI normally will not produce capitalization to emphasize words, which is, at best, archaic. It looks like an AI model quality issue.

Every token generated by an AI model is put through a random process of token selection, weighted by the certainty.

I would knock down the top:p being employed by the assistant to 0.9 or lower, to reduce not just the chance of low-quality tokens like temperature does, but to completely eliminate the tail of potential lottery-winning nonsense.

7

Thank you for the response - to be clear the capitalized words are expected behavior on my end (to denote specialized terminology, e.g. “Obstacle Character” is legitimate).