I have been added stuff in prompts, telling it to not answer, if Model does not know the answer.
But imagine, if I have complex information and AI model is also 50% sure if the answer it is outputting is correct or not.
How can AI tell me, if AI is happy with answer that its giving ?
There is unfortunately no one-size fits all approach here.
Especially when dealing with more complex queries, you should start with running some tests to identify for which queries model answers are either incorrect or vague. Depending on your set-up, you can then look at what might be causing this.
For example, if you have built a RAG pipeline that relies on similarity search to identify content relevant to answer, issues could be due to how you have chunked and embedded your content and/or how the user question that is used to perform the similarity search is phrased.
If the content extraction works well, then there could be issues with your prompt.
If you can be more specific about your use case and/or share examples of when you have encountered issues, we may be in a position to share additional guidance.