Assistants for Roleplay and reliably using retrieval

When using an assistant to manage a roleplay session, I often get hallucinations on informations I ask it about in a RP context.

For exemple, I have a PDF of my world’s info, including cities. When I ask, in-character, about the world’s cities, the response will always result in heavy hallucinations, giving name and descriptions that are not in the PDF although my instructions are very clear about not inventing new places and characters, and that everything that is mentioned should be something that is included in the reference file. Worst than that, it never mentions actual places that are mentioned in the file if it’s in a roleplay context.

But when I ask the assistant directly (ie. Hi Assistant, I think this character gave wrong info. Could you tell me what are the cities you know about and that are mentioned in your reference file), then it will mention the actual places correctly.

Is there any way to make the assistant more reliable on data retrieval and restrictions? Thanks!