I have a question, I am creating a chatbot, and I have filesearch, function_calling, and also in the prompt I tell it to respond in JSON, the problem is that I see that it does not always consult the file, and it is necessary that it always do so, and currently It’s not working, what should I do? thx!!! 
2 Likes
I have encountered a similar challenge.
In my experience, crafting a comprehensive system instruction that guides the model to consistently utilize the file_search tool proves effective.
Furthermore, providing the model with an understanding of the file structure and the appropriate details for utilizing the data retrieved through file_search is also beneficial.
7 Likes
Can you not just pass the file contents into the prompt, this is what I do.
I tried everything, in the file itself I also specify it, in the prompt and I even added an additional promopt to the user’s message, and even so it does what he wants, also one of the main problems is that, how to say, for each user it has than consulting a specific file and sometimes it fails too.
thanks for responding
1 Like
Could you please provide the current instructions, along with the temperature setting and an overview of the file structure?
2 Likes
Temperature : 0.55
Top p : 0.27
File : The files contain an extensive json with products so that the assistant knows the user’s catalog, since it is a chatbot to create invoices, products, quotes…
example content :
Please search for similarities in the product list automatically without calling other functions. [{“id”:“29413800”,“name”:“Abanico”,“fullReference”:“PRO00040”,“price”:0.0},{“id”:“29413816”,“name”:“Abanico”,“fullReference”:“PRO00056”,“price”:67.0},{“id”:“29413817”,“name”:“abaniqui8llo”,“fullReference”:“PRO00057”,“price”:79.0},{“id”:“29413790”,“name”:“Agua Mineral”,“fullReference”:“PRO00030”,“price”:0.8},{“id”:“29413837”,“name”:“azucar”,“fullReference”:“PRO00077”,“price”:50.0},
1 Like
In my opinion, specifying instructions directly within the json file is not an optimal approach.
Furthermore, the structure of this file can be optimized to enhance clarity, as the underlying model is still processing natural language.
Could you please elaborate on the type of similarity you are attempting to find, and the value of the instructions parameter? This information would assist us in understanding how the assistant deviates from the original instructions.
2 Likes
Wish I could help Cella! Personally, I just GO with the FLOW! Ergo I treat AI like being on a raft, with a pole, in a raging river! I am currently working on Toy Soldiers: Gamify multiplayer video game.
3 Likes
Are you on OpenAI-Discord? I asked your query to kapa.ai (ai-chatbot), perhaps there is something helpful in its answer, gl.
To address the issue of your chatbot not consistently consulting the file, consider the following steps based on the information from the knowledge sources:
-
Ensure Correct Tool Configuration: Make sure that the file_search tool is correctly configured and that the assistant is set up to use it. According to the OpenAI API documentation (https://platform.openai.com/docs/api-reference), the file_search tool requires a list of vector store IDs. Ensure that your assistant is configured with the correct vector store ID that contains your product data.
-
Model Compatibility: The file_search tool is specifically designed to work with certain models like gpt-4o. If you are using a different model, ensure that it supports the file_search tool. As noted in the forum post (https://community.openai.com/t/1041727), using the wrong model or tool name can lead to errors.
-
Structured Outputs and JSON Schema: If you are using function calling with JSON outputs, ensure that you have enabled structured outputs by setting strict: true in your configuration. This ensures that the model adheres to the JSON schema you provide, as mentioned in the function calling best practices (https://platform.openai.com/docs/guides/function-calling#best-practices).
-
Prompt Specificity: Make sure your prompts are specific and instruct the model to use the file_search` tool explicitly. According to the help article (https://help.openai.com/en/articles/9358033-key-guidelines-for-writing-instructions-for-custom-gpts#h_07bd5d91e2), providing explicit instructions for tool use can optimize performance.
-
Debugging Steps: If the issue persists, consider debugging by:
- Verifying the assistant and vector store configuration.
- Testing with the default event handler to see if the issue is with a custom handler.
- Inspecting raw responses for any error messages or clues.
- Simplifying the code to isolate the problem, as suggested in the GitHub issue (FileSearchToolCall.file_search has empty results · Issue #1966 · openai/openai-python · GitHub).
- Check for Updates: Ensure you are using the latest version of the OpenAI library, as updates may fix bugs related to tool usage.
If these steps do not resolve the issue, you may need to report it to OpenAI with detailed information about your setup and the problem. You can also refer to the OpenAI API documentation (https://platform.openai.com/docs/api-reference) for more detailed guidance on configuring and using the API effectively.
1 Like
What I did to solve the problem was passing and additional prompt with the user´s prompt specifying the name of the files in oder to get the assistant to always look into this files. Something like:
“”"
User´s Prompt: {user_prompt}
Make sure to consult this files in order to get and answer:
-Precios Catálogo.pdf
-Preguntas frecuentes.txt
-Sobre Nosotros.docx
If the answer is not in the files you may use the appropiate tool/function.
“”"
That should do the trick. Hope it helps
3 Likes
I did just that, I added an additional prompt to the user prompt and it still doesn’t work, do I have to be more specific?
In my experience, the more specific you are the better. Can you share what your additional prompt looks like?
1 Like
append("FILE_SEARCH: " + assistant.companyId + "_PRODUCTS, only check the catalog in this thread from that file; the user should not know anything about the other files.")
append("mini_prompt: ALWAYS respond in the specified JSON format, do not add unnecessary quotes or plain text. Also, consider the vectorFile to verify product names. DO NOT CREATE ANYTHING UNLESS THE USER REQUESTS IT. When responding, only include RECENTLY created items in the array of objects, not consulted or partially created ones. For consulted items, display them in the response_text.")
How does the final prompt looks like??
1 Like
FILE_SEARCH: 87734_PRODUCTS, only check the catalog in this thread from that file; the user should not know anything about the other files.mini_prompt: ALWAYS respond in the specified JSON format, do not add unnecessary quotes or plain text. Also, consider the vectorFile to verify product names. DO NOT CREATE ANYTHING UNLESS THE USER REQUESTS IT. When responding, only include RECENTLY created items in the array of objects, not consulted or partially created ones. For consulted items, display them in the response_text.message:HI! create one product with 5 stock 4 sales price
I just saw that you use 0.55 of Temperature, try to lower that down to something like 0.01. You should get more deterministic response. Furthermore, I think that maybe a catalog with almost 90k items is too much for the assistant to handle. What I would do is create a tool that can search in that file and return the matches for that particular query. When you do that you would give the assistant more specific information on what the user´s want. I do not know the structure of that file but if it is something like an excel file you should have no problem searching by column. Also, try to refactor that prompt into something like my previous answer. You should put the user´s prompt first and then maybe a list of the files names. I find that the assistant works better with that type of format.
1 Like
the file contains : [{“id”:“29413800”,“name”:“Abanico”,“fullReference”:“PRO00040”,“price”:0.0},{“id”:“29413816”,“name”:“Abanico”,“fullReference”:“PRO00056”,“price”:67.0},{“id”:“29413817”,“name”:“abaniqui8llo”,“fullReference”:“PRO00057”,“price”:79.0},{“id”:“29413790”,“name”:“Agua Mineral”,“fullReference”:“PRO00030”,“price”:0.8},{“id”:“29413837”,“name”:“azucar”,“fullReference”:“PRO00077”,“price”:50.0}, for example
Ahh I see, If the purpose of the file is for the user´s to search for an specific item, maybe you could make a tool that gets the name of the item and returns a list of all the matches.
1 Like
The idea was that the assistant would do exactly that, so as not to delay calls to my backend and fill my server with requests when it could simply look in the file and find the product/products that the user calls,
for example , the user prompt : create one invoice with 3 phones and 2 mouse (if the products exists in file, the assistant send the ids correctly in funtion call , If they are not found in the file, you can additionally call another function to look for it, and if it does not exist there it does not exist directly. and create new products)
In this particular case i think it´s better to leave that work to the backend. LLM´s have what is call a “Context Window” here you can get more information about what that is. But in simple terms, it means how much information a model can handle in a single run. For instance, if you wanted to know some specific thing that happened in a specific scenario in a specific chapter in all the Harry Potter books and you give the assistant all the books as it context, then it probably would have some troubles finding what you are looking for. Instead, if you only attach the chapter with the information you want, you would get more precise responses, probably in less time too. The problem that you are encountering is the same thing, it tries too look for a specific item in an array with 90k different items. You should try to simplify the task for it in order to get more meaningful responses.
2 Likes