How to fix Wrong Answer in Q&A

Hello everyone.
Now I’m developing Q&A system with custom knowledge base.
I have used Pinecone to embed vector and retrieve data.
I have embedded two documents, for example, financial statement pdf file of Google and Apple.
When question is “Which shareholders are common between Google and Apple?”, system generates wrong answer.
When asking about the shareholder of Google, it makes right answer.
In other words, individually it generates right answer but in combination wrong answer.
I have used gpt-3.5-turbo-16k model.
Please let me know the reason of this and how to fix it.


You should ask the model to create the search term to use, i.e. “Given this question ###{user_query}### what would be the ideal vector retrieval search prompt?” then use the result of that and vectorise it and search against that, then give the entire sequence, so including the first part and it’s answer along with the context of the vector retrieval and then ask the model to answer the original question, but this time for an answer, rather than a search prompt.

  1. Get User Input
  2. Ask LLM for best vector search to use for {user_input}
  3. Vectorise response
  4. Search for that vector and return the top 3-5 results
  5. Add all of that text to a new prompt and include a request to now solve the original {user_input} query given all of the above information.

Thanks for your quick reply.
I understand you.
But frankly speaking, now retrieved context to generate answer is enough to make exact answer.
So I think there is no need to use your method.