GPT 3 Semantic Search

Hello there,

Regarding the GPT -3 Semantic Search,
It seems there are two methods for uploading the content to API,

  1. via file id
    2. documents parameter as a string array

I find the semantic search gives a better result while using the documents parameter and it performs only lexical search while using the file id. Is there any specific reason for that?
Or is there any other improvisation i need to do while i pass it through the file id???

Please find the prompt for the file_id method.
{“text”:“I woke at 11 am and had my oats.”}
{“text”:“Oats is a healthy meal for breakfast.”}
{“text”:“Oats are made from wheat and rice.”}

API:
output = openai.Engine(“babbage”).search(
search_model=“babbage”,
query = ‘diet’,
max_rerank=5,
file = ppt_qa_fid
)
Output is :No related documents found.

Thanks

3 Likes

I’ve noticed that a longer search query/prompt produces better results when using the file method. But I don’t want to ask my users to write a longer query. Perhaps OpenAI should run a pre-analysis step (which they could charge for instead of providing free upload). The documents in the file could be analysed and grouped semantically, so that when a live search query comes in, there is a shorter and faster route to producing results. I agree with others who have posted on this that the keyword step is partially defeating the purpose of semantic search. Relatedly, OpenAI’s documentation is lacking in this regard.