How to ensure my chatbot stays within scope

Prompting
,
1

I have instructed my chatbot to only address queries that are in scope in the documents I uploaded in the vector database. If the user asks a question that is out of scope, it should not try to troubleshoot the problem.

For example, there is no document in the database regarding turning on one’s laptop. So if the first query is “How do I turn on my laptop” it provides the expected answer, telling the user that it is out of their scope.

" Used Retrieval

It seems there is no specific information available in the provided documents regarding troubleshooting a laptop that won’t turn on.

For further assistance, please contact IT support at support2@webmd.zendesk.com. They will be able to provide you with the necessary help."

However, when there is more context or a previous conversation, it will try to give a general troubleshooting.

How could I enforce the chatbot’s ability to stay within scope no matter what

2

get it to search you KB every time and take note of the cosine distance of the results.

if it drops below a certain threshold or there are no results, then provide an answer to the chatbot that directs it to state it cannot help the user?

you could consider strengthening that by dynamically changing the system prompt to re-inforce that.

or you could do some semantic routing from the off - if the embedding of the initial user query message diverges enough from a range of acceptable domains you could initiate a canned response that you can’t help the user with that query, without having to involve the Chat completion end point at all. That would also save you money.