How to fix Wrong Answer in Q&A

API
1

Hello everyone.
Now I’m developing Q&A system with custom knowledge base.
I have used Pinecone to embed vector and retrieve data.
I have embedded two documents, for example, financial statement pdf file of Google and Apple.
When question is “Which shareholders are common between Google and Apple?”, system generates wrong answer.
When asking about the shareholder of Google, it makes right answer.
In other words, individually it generates right answer but in combination wrong answer.
I have used gpt-3.5-turbo-16k model.
Please let me know the reason of this and how to fix it.
Thanks.

1 Like
2

Hi,

You should ask the model to create the search term to use, i.e. “Given this question ###{user_query}### what would be the ideal vector retrieval search prompt?” then use the result of that and vectorise it and search against that, then give the entire sequence, so including the first part and it’s answer along with the context of the vector retrieval and then ask the model to answer the original question, but this time for an answer, rather than a search prompt.

  1. Get User Input
  2. Ask LLM for best vector search to use for {user_input}
  3. Vectorise response
  4. Search for that vector and return the top 3-5 results
  5. Add all of that text to a new prompt and include a request to now solve the original {user_input} query given all of the above information.
2 Likes
3

Thanks for your quick reply.
I understand you.
But frankly speaking, now retrieved context to generate answer is enough to make exact answer.
So I think there is no need to use your method.

1 Like
4

When you say to vectorize the response and search for that vector, are you talking about having a vector database like pinecone that you’re using in conjunction with chatgpt?

I’m currently just uploading json files to the assistant.

1 Like
5

Yup, the assistants API abstracts away a bunch of the RAG pipeline, but for many of the use cases I deal with I need more granular control over that pipe. So I implement those steps, systems like Pinecone have their own assistants like API, but again, it’s removing fine control from the flow, so I opt out of it.

You can also make use of Vector DB’s like ChromaDB which you can just spin up loacally and use in a fairly lightweight footprint.

If you wish to take a bunch of the leg work out you can use framworks like LlamaIndex to handle most of the io and functionality while still keeping control over most aspects.

All depends on your use case and need for granularity.

2 Likes