Assistant API v2 / Revised answers for better future response

API
1

Hello,

I am wondering how to take care of something. We have build a website scraper which scrapes all webpages on a website into Markdown format. This pages are combined into a single .md file and uploaded to a vector store. This is the main information for our agent. The instruction on Assistant level is set that it can only answer questions related to information found in the knownledge base.

We also have build a dasbhoard where you can see what conversations have been going on, and what the assistant has answered. We have created a table to store “Revised answers” where we store the user input and the wanted response. All revised answsers are also stored in a markdown file format and uploaded to the file cluster.

Now here comes the problem. What content should be in;

  • The prompt/instruction
  • Files in the cluster
  • Perhaps fined tunings for custom model.

We now add markdown info the cluster files, but I dont have the feeling that the assistant is using these “Revised Q/As” very good in new conversations. Any tips for this?

2

I would split the scraped websites from the revised answers into a separate vectorstore collection. Then, I would either implement a bit more complex tool function that would first search collection with the revised answers. If no answer is found, then I would search the other collection.
Or better, I would use LangGraph to implement a “state machine” where I would implement/handle the logic when to search what and from where. I find LangGraph excellent if you are familiar with state-machines and you don’t have to use LangChain for the assistant/competition.

3

Hi Mark0, thanks for your suggestion. I was hoping to get this done using the OpenAI assistants v2 APIs only, and not have to build something completely new :slight_smile:

1 Like
4

Although, the tip with the seperated vectorstores does not seem that difficult btw

5

I fixed this by changing the prompt to always factcheck the response based on content in the files (files in the vector DB). Next to this, I also instruct the Assistants API to use the ToolsConstraint “file_search” as a required tool. This forces the Assistants API to always do a fetch of data in the vector database. If you do not give this hint, the Assistants API might respond without searching the vector DB.

This is how this looks in my C# code (using the C# OpenAI Beta SDK):

var runOptions = new RunCreationOptions() { AdditionalMessages = { content } };
runOptions.ToolConstraint = new ToolConstraint(new FileSearchToolDefinition() { MaxResults = 20 });

Works quite good now! and by using the GPT4o-mini model, it is quite cheap as well.

2 Likes