I had the same / similar question. I think this largely will depend on the specific implementation / desired outcome, but there is a question of when to add context from embeddings when there is a conversational aspect. For instance, with davinci (single shot), its more obvious: you simply provide the context with any question and get your answer. When we have a conversational paradigm, it becomes a little trickier. We can use the previously mentioned example:
User: “How much is this shoe?”
GPT: “The shoe is $5”
User: “Is it in blue?”
What would we send to the turbo model for this kind of conversation? I see a few options:
a) Perform an embeddings search for each user query separately and add it with context to the messages array, eg:
user: Please answer the following question with context: {Context 1}.
user: How much is this shoe?
assistant: The shoe is $5.
user: Please answer the following question with context: {Context 2}.
user: Is it in blue?
This approach is problematic because “is it in blue” is unlikely to return accurate semantic matches for the query alone.
b) Perform an embeddings search for all user queries combined (eg “How much is this shoe? Is it in blue?”) and include that context, eg:
user: Please answer the question using the following context: {Total Context}
user: How much is this shoe?
assistant: The shoe is $5.
user: Is it in blue?
c) Pre-process the user queries by prompting GPT Completion to summarize, eg:
Please summarize the following user queries and determine the ultimate question: "How much is this shoe? Is it in blue?"
And get such a response: How much is this shoe and is it in blue?. This can then be searched to determine embeddings context, and we can use the rest of the approach in b) to construct the final completions prompt.
So far I’ve had some success with the “B” approach. However, more complications arise with this approach if the user decides to switch contexts and talk about something else, eg. “What about this shirt?” In that case, the embedding query: “How much is this shoe? Is it in blue? What about this shirt?” may have more difficulty finding relevant documents for context. Using distance heuristics may be appropriate to see if the the embeddings result is “close” enough.
