Irrelevant sources being attached to responses

Hello everyone, I am working on bot which will respond to user entered queries. The bot will use a list of PDFs and content from various blogs stored in VectorIndex. Below is the code that I am using:

chat_engine = index.as_chat_engine(
        streaming=True,
        similarity_top_k=3,
        chat_mode="condense_plus_context",
        llm=llm,
        verbose=True,
    )
response = chat_engine.stream_chat(query)
sources = []
for node in response.source_nodes:
    sources.append(node.node.metadata["file_name"])

The issue that I am facing is that, the sources I get in response are not always relevant. So if I ask a question to this bot and it does not have information related to the question, it should ideally return empty list of sources but the sources returned are never empty and they contain at least one source every time

For those that wonder what is being shown here:

This code is likely using the LangChain library, which is designed for building applications with language models. The index.as_chat_engine method and the parameters used in this snippet are indicative of LangChain’s functionality for creating chat engines with features like streaming responses, similarity-based retrieval, and contextual chat modes.

Debugging 3rd party vector database operation isn’t really the forum’s forte.

You can at least instruct the AI agent that knowledge search may produce irrelevant results and that careful attention must be paid to return no citations at all if the results are not top-quality.