Hi I am new to LLM development, and I wanted some technical guidance or someone to suggest if there is something wrong with my approach.
I have a requirement where I have to create an AI agent that is able to interact with a custom tool that we have built ( that performs operations like normalization, clustering etc) and also if not part of the custom tool, be able to make a decision to use web search if it wants to search the latest information or also be able to generate code ( if user is asking for some simple ask like visualize this csv file ) ,
Currently I am planning to leverage the responses API using the Python SDK, because it has the in built web search and code interpreter tools for use and also have the agent connect to the custom tools (python files) that we have built. Would this be an appropriate approach ?
And also another question I had was whether I would be able to forward the files inputted by the user ( csv files, image files) to the LLM as part of the request ? because that would be necessary for code generation right ? I read that we can use the Files API to send our files but then not quite sure if this is feasible.
Your approach overall is sound. I would build the custom tool first (maybe more than one if the operations are super different; for example clustering has a lot more parameters than normalization etc.). Test the tools one at a time with direct invocation (ie not through the model) to make sure they work well as units. Don’t fall into the trap of trying to debug it all at once.
Then add your tools to a call that has the other (web, repl, etc.) tools you want to use from the platform and carefully work through edge cases of tool selection (ex: user says ‘find a group of values …’ could trigger a ‘search’ if the llm thinks its a web search, or a ‘cluster’ if the llm focuses on ‘group’) to refine the definitions of each tool. The model is very sensitive to the tool names and descriptions, especially when these are combined with other tools that you don’t control.
Your system/developer prompt, the tool names, tool descriptions, parameter names and parameter descriptions all matter… don’t try to test it manually and tweak things along the way: build an eval system that will run tests repeatedly and tell you what % of the time it is doing what you expect.
A couple of gotcha areas:
- it sounds like your custom tool needs access to data. Are you expecting the model to pass the data as a parameter or will the datasets be prohibitively large? I have had success telling the model “there is a dataset called my_car_data and another one called my_bus_data” and then making one of the tool parameters a “dataset name”. The model will choose the correct dataset name for your tool call, and the tool can go hit the data directly (database or duckdb or whatever) so the data itself is never in the model context (so faster and cheaper!). But if the model MUST have access to the data (ie to do a web search on the top 3 countries in the data or whatever) then experiment with different encodings for it (YAML, json, etc.)
- Regarding mixed modes like file uploads, you CAN always convert to text and stream contents into context (unless its just too large). OpenAI models are remarkably good at understanding whaver comes at them, but use the most appropriate API where possible and experiment with your specific use case.
Hey thanks a lot for this level of feedback, much appreciated.
with regards to the gotcha areas,
- Yes, for the custom tool, I am planning to do it like you said, using a local db itself and telling the model just the file name.
I am anticipating a use case like you said where the user uploads a csv file ( 1000+ rows) or more and asks the model to generate code to create some visual plots or for web search, would it be best to encode this data and send it along with the request ? because it is going to be too large right ? One possible option I was looking into was the Files API which allows us to upload files for the model to use, but it still seems like a hassle to send the entire large file across, would it be possible to just have the model generate the code and then I use the generated code to run locally based on the file names from my database, and if errors are thrown, send them back to the model to debug and fix ? Not sure how exactly I would do this though.
The “out of the box” design pattern in the response from @theshadowfight2modap is an approach much like one you are considering that will give you the capabilities you describe. Its well proven and reliable.
If the data is too large (or slow, or private) to upload, your alternative path (get a model to generate code and run it locally against the user data) will also work well but make sure you have a good reason to go this route as you will have more work to do yourself (when the model sends you code, you have to set up a sandbox to run it with appropriate hooks for the data access functions etc.) and you won’t have the advantage of future enhancements that OpenAI makes in their pattern.
For either of these cases, i can recommend some best practices for getting the model to do useful things with your data:
- gather business context from the user if you can. What is the context of the upload? personal finance? sports scores? Do they have a strategy or a website that describes what they are trying to do? Its all valuable.
- Using traditional code, look at the data and create a text description as best you can. For dataframes (csv, yaml, json, etc.) provide a list of columns and the types of data in those columns, list of most common values in each attribute column, statistical max/min/average and null count etc.) for metric columns. When you can’t list all the values of a field (ie 10,000 names) just tell the model “the name field is a customer name like ‘name 1’ or ‘name 2’ among many possible values. When there are only a handful of values, tell the model “the gender column is either ‘male’ or ‘female’”. You can determine all of that with highly deterministic fast procedural code and it is absolute GOLD for the model to use. Include a few sample rows (I recommend using json encoding for OpenAI models). IF you have really wide tables (over 200 columns) I would recommend narrowing the scope of what you include for the model to consider. One way to do that is to write a preparatory query like ‘the user wants to count the number of classrooms with more than 20 students. Can you tell me which of the following 500 columns would be needed by a DBA to make that determination? respond with a comma separated list of column values between <column_list> and </column_list>…”
- include both of those (the business context and the metadata) when you ask the model to write code against it.
This is an active area of development that is approaching the NL2SQL problem or “natural language query”. There’s a universe of activity in academia and industry, including a dedicated benchmark https://spider2-sql.github.io/
Mike