Follow-up questions in a QnA environment

API
1

I’m working on a QnA system for the business I work at. We have a big knowledge store of long documents, that change every now and then. I’ve build a script to split the documents in smaller documents (up to around 1000 tokens) then, using the embeddings API, I create the embeddings and store them. Then the user asks a question, which is also embedded and then use the cosine similarity to find the appropriate top k text fragments and call the completion API. So far so good.
However, I was wondering, how would I create a system where the user can either ask follow-up questions, where the question won’t have enough context and consequently the embedding-search mechanism wouldn’t work on the question alone, or ask questions on completely different topics.

I’ve thought of maybe summarizing the previous context and adding it to the prompt but I fear that could ‘harm’ the search if the question is on a different topic.

My question is, what is the best way to approach this problem? Multiple API calls are not and issue, and fine-tuning wouldn’t be a problem either.
Thanks.

1 Like
2

Hi @apavia I am working on something very similar and I was thinking about passing to the embeddings-search function a modified query. I think it is safe to assume the user will do mostly follow-up questions, since it is simply possible to create new “conversations” to change the topic radically. With this in mind, my first idea would be to append the previous n queries to the current n+1 query. I’ll give it a try shortly, let me know if you did something else and how well it went :smiley:

3

I ended up appending also part of the assistant responses to the subsequent inputs, since sometimes the content of the follow up questions is actually given by the AI assistant responses. So far that makes the retrieved “chunks” to be more “accurate”, according to my quick tests.