I’m trying to access a json file with 56 objects each with key/values that i want my assistant or response api to query for specific user selection inputs as well as short text inputs that I’d like for the model to reason and find the best 3 best objects and their key/values even if the inputs don’t contain any keywords. within my prompt i highlight that we should always out data from my json file and include examples.
Initially I tried using file search for this but when i accessed the endpoint from my app i didn’t get an object from my file as expected, The model tried derive own data.
For me get a consistent output: object given the user inpout i’d be nice if the LLM could reason in this step what approach should i be using for this ? What type prompt, or approach structured outputs, RAG, JSON mode?
You have data that already has individual entities, and potentially a database-like format that can be validated.
You have two main options:
load data to such a database of fields that can be queried and searched;
extract the unique natural language of objects, and use embeddings for semantic search.
Then for presentation to an AI:
a function that the AI can call, surrounded by a task that encourages AI to write queries and utilize the tool for answering;
an automatic injection of knowledge to answer from.
Then you can utilize the intelligence of the language AI to:
style the output consistently, by instruction, by examples;
answer directly from that knowledge in the style and quantity needed;
refuse to answer without having enough sourced knowledge.
Structured data, tabular databases, CSV or JSON, of a size beyond loading it all into the AI context as a message, does not work well with any offered technology or internal tools. You develop your own.
Adding to this, since MCP is now supported as first class citizen, you should just find a database that has MCP support, dump your data there and let the model just use the function calls.
You can always make your own for maximum flexibility and control but there are a lot of really good open source solutions for this already.
No! That’s not good architecture, and this is distraction from the original question.
MCP is a reasonable choice if you are presenting a SaaS service (“weather API”, “market data API”), but if you just want to query your own database, just write a function and connect to the database with whatever library you normally do that with.
Why on earth would you go as far as exposing your own data on the interweb behind https (and all the additional engineering that requires to serve it securely) just so you can query your own DB?
Underlooked that point. My perspective was for a public/authorized service (like handling events reservations via chat with AI where the chatbot authorizes the user within the app serving the data).
But if it is for internal use you definitely could skip the MCP server part and just provide tools.