GPT-4.1 no annotations from vectorstore in answer via assistant api

API
,
1

GPT40 returns annotations for each source document in the vector store that is attached to it, I’m not seeing any annotations when I switch the model to use the new GPT-4.1 model.

I’ve tried tweaking the instructions a bit but no success so far. Is this a bug or new way of having to tell it via system instructions to include annotation citations for any vector store data source used in the answer?

2

In assistants, gpt-4.1 and gpt-4.1-mini are getting the myfiles_browser tool loaded on it, instead of file_search of gpt-4o-2024-08 and newer.

This means if you are talking about the wrong thing, you will get the wrong action.

The image is a description of the "myfiles_browser" tool, explaining its function for searching files in a conversation using Python syntax and guidelines for effective usage. (Captioned by AI)
The image is a description of the "myfiles_browser" tool, explaining its function for searching files in a conversation using Python syntax and guidelines for effective usage. (Captioned by AI)592×540 20.8 KB

Having it write annotations in the playground is successful, and they get hit with the playground’s annoying rewriting preventing diagnosis of true outputs.

A user requests information about a minimal document, while the ChatAPI reveals the file as a placeholder with no substantial content. (Captioned by AI)
A user requests information about a minimal document, while the ChatAPI reveals the file as a placeholder with no substantial content. (Captioned by AI)593×342 12.1 KB

The run steps has the file index used in that in-document citation in [1]step_details->tool_calls->file_search → results.

3

so I’m grabbing them from here, has it changed?

{
  "id": "msg_123",
  "object": "thread.message.delta",
  "delta": {
    "content": [
      {
        "index": 0,
        "type": "text",
        "text": { "value": "Hello", "annotations": [] }
      }
    ]
  }
}
4

Annotations has never worked great, but you should have and see the fancy japanese brackets in your output text or there certainly won’t be anything extracted by the endpoint.

I would expect the same pattern as gpt-4o-mini in assistants which uses the same internal tool name. It is possible that nobody on the inside (like nobody serious on the outside) is paying attention to assistants - and this model is 10 hours old.

1 Like
5

yeah I know they are deprecating assistants eventually in favour of responses, but currently they aren’t feature complete so I can’t make the switch yet. I hope someone at openai reads this thread and keeps support for Assistants even in newer models until there is full feature duplication in responses, otherwise I can’t use the new models!