Modifying assistant's retrieval file using function calling but it doesn't recheck its retrieval file before answering

API
1

Hello, I am back with another GPT assistant in the Unity Editor problem.

I am using function calling to allow the assistant to find relevant scripts in my Unity project and add them for retrieval. I then cancel the current run, modify the assistant to have the new knowledge, then run the assistant on the thread again. Everything seems to work as it should, but the assistant does not take into account the updated retrieval file; instead it will make up the contents of the script that was just added when attempting to answer the question about the script.

More details:
I am just using one retrieval file that consists of a detailed project structure, which gives the filepaths of every file and folder in the project, followed by any script contents from the project that I want it to start with.

I have a list of text assets that include any scripts I want to include in the upload, plus the projectStructure file. The projectStructure file gets updated anytime I update or create an assistant.

What essentially happens is, in its instructions I tell it to reference the main retrieval doc for any questions related to the project, and if it doesn’t have any data on whatever the script the user is referencing, it should search the projectStructure for the script, and return the filepath with the function “add_script”. That script then gets added and the retrieval document gets updated and reuploaded. Then I cancel the current run so that I can update the actual assistant with the new retrieval file. Then the same assistant gets run on the same thread.

What I think should be happening, is when the assistant gets called again to run on the same thread, it should be again going through the process of checking the retrieval document for relevant information and answering according to that, but it seems to just make up the script content. I can verify that it does in fact have the actual data by just asking it if it actually looked at the retrieval file. It will apologize and say that it had a misunderstanding and proceed to give me the correct information about the script.

Is there something I can add to the flow in order to make sure the assistant checks its knowledge when I re-run it?

1 Like
2

When running the Assistant again, you have two options during the Run Thread step
a) Modify instructions - where you can give it more files etc
b) (what I think you should do) Update Instruction - where you are replacing the instruction again (with the same base instruction and files)

4

So, I do re-set the assistant instructions and re-upload the file. I even have added a hard coded addition to the prompt instructing the assistant to check the retrieval file before answering. It still makes up the script content that was uploaded that it does have access to. I asked the assistant about this and got this response:

"I understand the issue you’re pointing out now, and I’d like to explain what went wrong in my initial response regarding the TransformSequence.cs content. The error occurred due to an oversight in the process of retrieving and presenting the script content correctly after adding it to my knowledge base.

Here’s a closer look at the process and where the breakdown happened:

  1. Adding Script for Analysis: After you uploaded the ProjectStructureAndScripts.txt and requested information about TransformSequence.cs, I used the functions.add_script function with the filepath provided to scan through the document for the script’s content. This step was to integrate the specific script into my knowledge base for analysis.

  2. Generating Response: When generating the response, instead of relaying the precise content found in the document through the addition mechanism, I incorrectly presented generalized information. That is, the content of TransformSequence.cs I provided was not extracted directly from the added script but was constructed based on a conceptual understanding of what such a script might contain. This was an error in communication and processing on my part.

  3. Overlooking the Correct Script Content: Though I had the means to access and incorporate the precise contents of TransformSequence.cs from your document into my response, I erred in not directly relaying that accurate information to you. This resulted in providing content that did not reflect your actual script.

The mistake here lies in the execution phase where the specific details obtained from the document should have been the basis of my response about the script. Instead, I deviated by providing a hypothetical version that was not aligned with the actual content, leading to confusion and inaccuracies in addressing your query correctly.

I hope this clarifies the error in my approach. Going forward, I will ensure to rely strictly on the accurate and specific data provided to respond to your inquiries more precisely."

I have even tried entirely deleting the assistant and creating a new one to run on the same thread but it still has the same issue. But if I tell it that it is wrong, it will come back with a correct response. It is so weird.

1 Like