Assistant with a file_search tool not working

I’ve been struggling to make the assistant work for a few evenings already. At this point, I am out of ideas and would really appreciate any help. I am following the Assistant Quick Start Guide to set up an assistant that responds with information based on an uploaded file. However, even the simplest setup results in a gibberish, made-up message that has nothing to do with the uploaded file.

I have uploaded a PDF file via the Files API. The file is approximately 600 KB of text with no images. Here is the response from the API:

 {
  "object": "file",
  "id": "file-AUuWQg6dWh9gHgBedgwd1x",
  "purpose": "assistants",
  "filename": "872dff39-9a74-4cdf-b150-9fe123625857.pdf",
  "bytes": 624031,
  "created_at": 1737460874,
  "status": "processed",
  "status_details": null
}

I also have a vector store created and ready (the file has been added and is in a complete state):

// Upon creation: 
{
 "id": "vs_epzDGLAlNaZznY4CWhE5dibv",
  "object": "vector_store",
  "name": "test_vstore_2",
  "status": "in_progress",
  "usage_bytes": 0,
  "created_at": 1737460875,
  "file_counts": {
    "in_progress": 1,
    "completed": 0,
    "failed": 0,
    "cancelled": 0,
    "total": 1
  },
  "metadata": {},
  "expires_after": null,
  "expires_at": null,
  "last_active_at": 1737460875
}

// some time later, before I proceed I make sure the store is "ready"
{ 
"id": "vs_epzDGLAlNaZznY4CWhE5dibv",
  "object": "vector_store",
  "name": "test_vstore_2",
  "status": "completed",
  "usage_bytes": 1842,
  "created_at": 1737460875,
  "file_counts": {
    "in_progress": 0,
    "completed": 1,
    "failed": 0,
    "cancelled": 0,
    "total": 1
  },
  "metadata": {},
  "expires_after": null,
  "expires_at": null,
  "last_active_at": 1737460875
}

I have an assistant created with the FileSearch tool enabled, along with tool resources referencing the vector store. The assistant’s prompt is a desperate attempt to get something meaningful to see if it actually reads the file contents:

// api response:
 {
  "id": "asst_suT4vxw8R4ps30KhluxnPJlN",
  "object": "assistant",
  "created_at": 1737460877,
  "name": "Test accountant",
  "description": null,
  "model": "gpt-4o",
  "instructions": "You are a viewer of the files. You check the file if a message contains file reference and you describe what you see in that file. Your response should be in a json format, where there is just one message field with free text describing the content of the file",
  "tools": [
    {
      "type": "file_search",
      "file_search": {
        "ranking_options": {
          "ranker": "default_2024_08_21",
          "score_threshold": 0.0
        }
      }
    }
  ],
  "top_p": 1.0, // defaults
  "temperature": 1.0, // defaults
  "tool_resources": {
    "file_search": {
      "vector_store_ids": [
        "vs_epzDGLAlNaZznY4CWhE5dibv"
      ]
    }
  },
  "metadata": {},
  "response_format": {
    "type": "json_object"
  }
}

I also have a thread with one message containing an attachment:

 {
"id": "thread_ZG7TeIeaGF8IjijbbxQaENKs",
  "object": "thread",
  "created_at": 1737462230,
  "metadata": {},
  "tool_resources": {
    "file_search": {
      "vector_store_ids": [
        "vs_epzDGLAlNaZznY4CWhE5dibv"
      ]
    }
  }
}

Then I create a run and wait for its completion:

{
  "id": "run_BoWZXnbuYw9veZecADNGRNUr",
  "object": "thread.run",
  "created_at": 1737460883,
  "assistant_id": "asst_suT4vxw8R4ps30KhluxnPJlN",
  "thread_id": "thread_ZG7TeIeaGF8IjijbbxQaENKs",
  "status": "completed",
  "started_at": 1737460884,
  "expires_at": null,
  "cancelled_at": null,
  "failed_at": null,
  "completed_at": 1737460886,
  "required_action": null,
  "last_error": null,
  "model": "gpt-4o",
  "instructions": "You are a viewer of the files. You check the file if a message contains file reference and you describe what you see in that file. Your response should be in a json format, where there is just one message field with free text describing the content of the file",
  "tools": [],
  "tool_resources": {},
  "metadata": {},
  "temperature": 1,
  "top_p": 1,
  "max_completion_tokens": null,
  "max_prompt_tokens": null,
  "truncation_strategy": {
    "type": "auto",
    "last_messages": null
  },
  "incomplete_details": null,
  "usage": {
    "prompt_tokens": 129,
    "completion_tokens": 37,
    "total_tokens": 166,
    "prompt_token_details": {
      "cached_tokens": 0
    }
  },
  "response_format": {
    "type": "json_object"
  },
  "tool_choice": "auto",
  "parallel_tool_calls": true
}

However, I always receive a response containing gibberish data that is clearly generated without checking the file contents:

 {
      "id": "msg_fWrWhqCfNjeSz7Ll4gXxZrGb",
      "object": "thread.message",
      "created_at": 1737460885,
      "assistant_id": "asst_suT4vxw8R4ps30KhluxnPJlN",
      "thread_id": "thread_ZG7TeIeaGF8IjijbbxQaENKs",
      "run_id": "run_BoWZXnbuYw9veZecADNGRNUr",
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": {
            "value": "{\"message\":\"The attached file consists of a worksheet detailing a list of transactions with columns labeled \\\"Name,\\\" \\\"Transaction ID,\\\" \\\"Date,\\\" \\\"Amount,\\\" and \\\"Transaction Type.\\\" Here are some of the contents observed:\\n\\n1. **Names**: The list includes names like John Doe, Jane Smith, and Mike Brown.\\n\\n2. **Transaction IDs**: There are unique alphanumeric identifiers such as TXN12345, TXN54321, and TXN67890.\\n\\n3. **Dates**: Transactions are dated from \\\"01/01/2022\\\" to \\\"05/05/2022.\\\"\\n\\n4. **Amounts**: Monetary amounts are recorded, for example, $100.00, $250.50, and $75.25.\\n\\n5. **Transaction Types**: Types include descriptors like \\\"Credit,\\\" \\\"Debit,\\\" and \\\"Transfer.\\\"\"}",
            "annotations": []
          }
        }
      ],
      "attachments": [],
      "metadata": {}
    }

I’ve tried different messages (meaning the prompt; the result is always trash, as it clearly doesn’t read the file content), various models, and even creating the assistant first and then updating it with a tool and tool resource. No luck so far. To me, it looks like the API is just broken. There is nothing special about the file itself, and the web UI version works absolutely fine with it, providing all the requested information.

At this point, I am not sure what else to check. Any help will be highly appreciated.

1 Like

I really need a rubber duck at this point. My serialization of the create run request was wrong and I’ve overridden the tool for the assistant with an empty array, thus disabling the file search. It’s funny I’ve noticed it only after posting the message here. Works good now.

2 Likes