I am making a programming assistant, where project’s files are included in file_search tool. However it doesn’t like to use it. Unless I specifically ask it to describe the project it will perform file_search, but in any other scenario it will just randomly start generating code (often in a different language altogether!).
Is there some instruction prompt that can help? I’ve tried “Before doing any operations, use file search to better understand the the project.” but doesn’t seem to help.
To enforce the use of a tool like file_search, you can use tool_choice on the Assistant runs/createRun API:
Specifying a particular tool like
{"type": "file_search"}or{"type": "function", "function": {"name": "my_function"}}forces the model to call that tool.
just what I needed, thanks!
looks like specifying tool_choice prevents assistant from using any other tools sadly. Now it’s ONLY using file_search and none of the other tools. I hoped there was better API for specifying tool priority.
Also why is it in the Run object, should really be in Assistant object tools… something like
"tools": [
{
"type": "file_search"
"weight": "1.0"
},
{
"type": "function"
"weight": "0.8"
"function": { ... }
}
],
The /assistants endpoint manages Assistant objects. The /runs endpoint handles interactions with such an Assistant.
A weighted tool_choice would give more room to play with tools. It’s difficult to know how weight would work though: would it be a sampling?
Yeah I understand that. My point is more of general workflow (at least from my perspective). Assistant has tools it can use. So it makes sense to have some sort of tool preference set up in assistant object as well instead of run object. Run should just execute a thread with assistant, if I want to change assistant behavior I would make changes to assistant, not the run. Why put tool_choice in run if assigned assistant might not even have that tool?
I am not very knowledgeable in this area. Maybe weight alone is not sufficient. At least in my project, I’d like to specify couple things: required tools to call before all other tools (but it should not prevent assistant from calling other tools). And some sort of tool preference setting - I don’t know whether it should be weight or priority or importance or something.
e.g. in my project (programming assistant) what I’d like to see, when user asks a question:
file_search firstget_active_file_text next.get_selected_text next.I can get this workflow working with prompt engineering and setting specific assistant instructions. But it’s not 100% consistent, and I would like to reduce amount of text required from the user to complete the task.
Assistants requires exposure of the underlying features in Chat Completions AI models in order to perform such things. In this case, the model presents no feature to decide bias of calling functions, and direct suggestion of a control over what they don’t want controlled was met with silence.
Whether an AI model calls a function is usually the first thing out of its digital mouth. The AI has decided within max_tokens=1 whether it wants to respond to a user or emit to a function. And OpenAI deliberately nerfed logit_bias so you couldn’t affect this.
Here is an example of my AI assistant with a single custom tool vs without any custom tool. Both have file_search with the same vector store. The only difference is me adding CreateReadme tool.
With CreateReadme tool and file_search:
Without any custom tools besides file_search:
So it clearly refuses to use file_search if there are any custom tools added. The issue is I need it to use file_search prior to doing anything else.
So looks like it may be dependent on the model? In the above post I’ve used “gpt-3.5-turbo-0125”
I’ve tried it with “gpt-4o” and it looks like it’s actually calling file_search properly.
I’ve done multiple repetitive tests with both models, and 4o works consistently, while 3.5 doesn’t work at all.
I am in agreement that GPT-4o is considerably more reliable with some prompting on triggering tool calls.
I will also add that if you want to you could add some deterministic rules as well as create multiple assistants. One for the start of your interactions and after tool call file_search is triggered you could hand it off to an assistant set up with different tools and different prompts.
I am currently leveraging different assistants for different tasks and roles in order to ensure more graceful flows during the conversation.
That particular like of thought would not work.
Once you invoke a run, the agent that is an “Assistant” will only interact with you by your own functions you provide. OpenAI’s tools, such as the file search or code interpreter, are sent to and return to the assistant internally, and the AI is automatically invoked again…and again. Until finally placing a response viewable as a thread message. Therefore a different AI assistant cannot answer about the file search results than the one that invoked it, unless you ask again.
I’m thinking if you use chat completions, and just employ assistants for its file search because you haven’t set up any other real database…what do you think?
