Retriever / reader pattern

Hi Guys
I’m playing around with semantic search in a way that I embed a lot of text paragraphs. Then embed a question, find close documents on wich basis Gpt-3 then formulates the answer. I am struggling with one point though : how do I decide what documents are „close enough“ to be considered to be part of the answer ? I can say „give me too 6 docs“ or check a max distance from my question - but I never know wether one more document would also have been relevant.
Any ideas how to deal with that ?