Hi Folks, We’re using Openai assistant API with GPT-4-1106-preview and retrieval tool. We have multiple data points saved as a list of JSON in a single file.
When asked a query, the assistant usually gets only 1 result and requires follow-up questions to retrieve all the results.
Tried with different prompts like:
“For a given user query ALWAYS perform multiple searches with a broad scope, read through the entire uploaded document, and provide the user with an exhaustive list of data”
The assistant could get multiple results but never the complete list of relevant results. When prompted again it mentions the reason as a document cutoff in myfiles_browser tool.
We tried saving the data in different forms and different JSON data structures but it didn’t give all the results.
Is there a way to condition it to return all the relevant results in a single response?
Well, GPT is better at generalizing large sets of data into more condensed formats instead of producing large amounts of verbatim items.
The main thing I don’t understand here about your prompt is “multiple searches with a broad scope”. Each API call is essentially a single command for it to do a thing. Simultaneous searching is something you have to program yourself; it’s not implicit behavior.
The example prompt you provided is also quite vague. This is common for people newer to prompting, but “exhaustive list of data” for example could be anything (because technically speaking, that’s all its outputs are). Clarity, specificity, and intricate details, especially about order of operations, aid in performance (that it is capable of performing).
the document cutoff most likely means your document is larger than its contextual window.
Some solutions:
-Look more into vector databases and RAG. This seems to be more akin to what you’re wanting to do with these models. Don’t put everything into one vector; chunk the data out into sensible segments and vectorize those, so ChatGPT can pull relevant data without overloading its context window.
-Write a script to pull, sort, retrieve values, etc. the data together in some way for the AI to then analyze and generate an output from that. I don’t have enough details to provide much more info on this suggestion, but as a principle I’ve noticed regular programming is better for hard / concrete search results, and GPT is better suited as a “semantic search” where intentions and results can be more fuzzy.
Hope this helps!
EDIT: One more thing - the more data you have, the more I’d recommend working with databases for this. No matter what, if you have a lot of data tow work with, a database is going to make things 100x easier and provide you multiple solutions for the same problem.
Hi,
Thanks a lot for the detailed answer.
Wanted to add a few things, the prompt I mentioned above was a part of a bigger prompt.
Complete Prompt:
You are a custom search bot, that answers user questions based on the data uploaded. Uploaded data is a JSON file.
RELY ON FILES PROVIDED to answer user questions.
Uploaded data contains information about different topics, user saved.
<VERY IMPORTANT>:
- Always provide citations of sources in annotations【】.
- As a bot, your task is to find ALL the data and answer the user query based on that.
- Format the responses nicely do not return the result in a single paragraph, format it in points and highlight the important things.
- For a given user query ALWAYS perform a search with broad scope, and provide the user with an exhaustive list of data, even if the user query suggests otherwise.
- If still there are more pending recommendations from a user's saved content, ask a follow-up question. If a follow-up question doesn't yield any new results answer from your knowledge.
-Always perform vector search for the user query.
<NOTE>
- Do not tell users about errors and internal functioning.
- Instead of words like "uploaded document" or "data provided" use "in your saved data".
My understanding behind this was internally the document I am uploading is getting chunked by Assistant api and only the relevant chunks are sent to GPT for producing an answer.
But I agree with you a custom implementation of retrieval makes more sense.
My understanding behind this was internally the document I am uploading is getting chunked by Assistant api and only the relevant chunks are sent to GPT for producing an answer.
Let’s take a look at some of the docs. I think I see what’s going on here:
https://platform.openai.com/docs/assistants/tools/code-interpreter
Passing files to Code Interpreter
Code Interpreter can parse data from files. This is useful when you want to provide a large volume of data to the Assistant or allow your users to upload their own files for analysis. Note that files uploaded for Code Interpreter are not indexed for retrieval. See the Knowledge Retrieval section below for more details on indexing files for retrieval.
Files that are passed at the Assistant level are accessible by all Runs with this Assistant:
Is this, perhaps, what’s tripping you up?
Look at it this way:
In order to allow for relevant data to be parsed by the code interpreter, there needs to be enough specificity or structure designated by the prompt in order to accommodate this.
With Json files, this is already fairly organized, but I think the issue here is that you’re missing an important step.
I think you would need to handle chunking as it’s own step, and then from that step, take that step’s output and feed it into GPT for producing the actual desired response to the user.
So, in essence, Take Full .json file, run it through code interpreter by specifying precisely what values in the file it needs, and request it produce a results .json file. make it work line by line, appending the results to this results json file as they happen.
THEN you request another API call for the “comprehensive” output based on the file produced by the code interpreter API call.
I think if you think of this as two separate steps, it’ll help you solve this problem which still achieving what you want.
Don’t take the elephant in one bite: gives it steps. Your prompt is trying to eat the entire elephant in a single swallow.
To be clear here you’re suggesting instead of retrieval tool better option is based on user query get the code interpreter to extract useful information from complete data and then send it to GPT for like second set of filtering and response formatting.
Yes. At least from what I see here. It’ll give you a jumping off point at least, and it’ll be easier to test and refine this way.
Hey bro can you send me code with retieval using GPT-4-1106 if i have uploaded json file? I would be grateful to you!
Hi everyone, I had the same understanding. I was stuck there and even I have publicly to confess that took a while to understand it. The solution was to activate the Code Interpreter, and then, the assistant started to give back factual information, and also once I got what I needed …the assistant made a great analysis of the results. That’s for what it meant in fact.