Can the Retrieval function be used accurately according to the requirements in the prompt project?

When I ask questions that do not involve my special books, I want GPT3.5 to answer them. If the user asks some special questions, I want to use the books in the knowledge base to search. , the current situation is that when I ask some irrelevant questions, it is still answering book-related questions or has not found the answer. Can the prompt project make accurate judgments in this regard, or can it only perform fine-tune work?