Make assistant use file_search more pro-actively

I am making a programming assistant, where project’s files are included in file_search tool. However it doesn’t like to use it. Unless I specifically ask it to describe the project it will perform file_search, but in any other scenario it will just randomly start generating code (often in a different language altogether!).

Is there some instruction prompt that can help? I’ve tried “Before doing any operations, use file search to better understand the the project.” but doesn’t seem to help.

To enforce the use of a tool like file_search, you can use tool_choice on the Assistant runs/createRun API:

Specifying a particular tool like {"type": "file_search"} or {"type": "function", "function": {"name": "my_function"}} forces the model to call that tool.

1 Like

just what I needed, thanks!

looks like specifying tool_choice prevents assistant from using any other tools sadly. Now it’s ONLY using file_search and none of the other tools. I hoped there was better API for specifying tool priority.

Also why is it in the Run object, should really be in Assistant object tools… something like

"tools": [
      "type": "file_search"
      "weight": "1.0"
      "type": "function"
      "weight": "0.8"
      "function": { ... }
1 Like

The /assistants endpoint manages Assistant objects. The /runs endpoint handles interactions with such an Assistant.

A weighted tool_choice would give more room to play with tools. It’s difficult to know how weight would work though: would it be a sampling?

Yeah I understand that. My point is more of general workflow (at least from my perspective). Assistant has tools it can use. So it makes sense to have some sort of tool preference set up in assistant object as well instead of run object. Run should just execute a thread with assistant, if I want to change assistant behavior I would make changes to assistant, not the run. Why put tool_choice in run if assigned assistant might not even have that tool?

I am not very knowledgeable in this area. Maybe weight alone is not sufficient. At least in my project, I’d like to specify couple things: required tools to call before all other tools (but it should not prevent assistant from calling other tools). And some sort of tool preference setting - I don’t know whether it should be weight or priority or importance or something.

e.g. in my project (programming assistant) what I’d like to see, when user asks a question:

  1. call file_search first
  2. call get_active_file_text next.
  3. call get_selected_text next.
  4. call other tools as needed to complete the run.
    (2 and 3 could be merged together into single tool, but I am just trying to explain my point - assistant should first try to get some awareness of situation before performing a task).

I can get this workflow working with prompt engineering and setting specific assistant instructions. But it’s not 100% consistent, and I would like to reduce amount of text required from the user to complete the task.

1 Like

Assistants requires exposure of the underlying features in Chat Completions AI models in order to perform such things. In this case, the model presents no feature to decide bias of calling functions, and direct suggestion of a control over what they don’t want controlled was met with silence.

Whether an AI model calls a function is usually the first thing out of its digital mouth. The AI has decided within max_tokens=1 whether it wants to respond to a user or emit to a function. And OpenAI deliberately nerfed logit_bias so you couldn’t affect this.

Here is an example of my AI assistant with a single custom tool vs without any custom tool. Both have file_search with the same vector store. The only difference is me adding CreateReadme tool.

With CreateReadme tool and file_search:

(creates readme with some random nonsense, clearly not using file_search)

Without any custom tools besides file_search:

(writes actual project description in chat)

So it clearly refuses to use file_search if there are any custom tools added. The issue is I need it to use file_search prior to doing anything else.

1 Like

So looks like it may be dependent on the model? In the above post I’ve used “gpt-3.5-turbo-0125”
I’ve tried it with “gpt-4o” and it looks like it’s actually calling file_search properly.

I’ve done multiple repetitive tests with both models, and 4o works consistently, while 3.5 doesn’t work at all.