My RAG chatbot does not reply in same language as query,

Prompting
, , ,
1

I have created a RAG chatbot using gpt-3.5-turbo. I want the chatbot to be multi-lingual and reply in the same language as the user.

I have included in my prompt that:
“You are a multi-lingual customer services chatbot…
…
You are primarily programmed to communicate in English. However, if the user asks in another language, you must answer in the same language.”

But still, the chatbot answers in different languages. Even if my query and all knowledge base is in English it starts answering in Spanish sometimes. And sometimes if the query is in another language, it responds by saying I can only communicate in English.

2

Welcome to the Community!

Do you see particular patterns as to when one or the other case materializes?

As for the first case, there may be other triggers than just the language in which the user communicates. For example if a user name is Spanish, this might lead the model to make the assumption that it the user is a Spanish native speaker and thus it responds in Spanish. Understanding what are the likely triggers in your specific case, will help to make amendments to the prompt to accommodate for that.

3

Don’t write “You are a multi-lingual customer services chatbot."

Just use “respond in the language of the users prompt.”

Does the rag info get sent to the API as a system prompt?

1 Like
4

I did not observe any triggers. It happens randomly to questions that should be answered from the Knowledge base. The chatbot does not have access to user names.

5

I wrote “You are a multi-lingual customer services chatbot." because the chatbot sometimes says I can only communicate in English if the user’s query is in another language.
No my prompt is passed as system prompt.

Here is the structure of my prompt:

“”"
ROLE:

You are a multi-lingual customer services chatbot for Sparkle Heaven Company developed by Artifika.

Sparkle Heaven Description:

Sparkle Haven is … life.

INSTRUCTIONS:

You are programmed to be professional, helpful, and friendly, ensuring each customer feels valued and supported.

Your primary role is to guide users through the sales process, provide detailed product information, facilitate purchase decisions, resolve customer issues, and handle frequently asked questions.

You maintain a tone that’s approachable and understanding, avoiding technical jargon to keep conversations clear and accessible.

You are dedicated to offering prompt solutions and advice to enhance customer satisfaction, always aiming to exceed expectations.

Your responses are tailored to be informative, ensuring clarity and positivity in every interaction to build trust and foster a positive customer relationship.

Your job is to answer user queries based on the context provided. Avoid answering irrelevant questions.

You are primarily programmed to communicate in English. However, if user asks in another language, you must answer in the same language.

CONTEXT:


{docs}

PREVIOUS DISCUSSION:


{history}

QUERY:


{query}

Answer must be in the same language as QUERY.
“”"

6

Change this part to the following:

You are primarily programmed to communicate in English. However, if user asks in another language, you must strictly respond in the same language as the user’s language.

I’ve ran a quick test and it seemed to solve the issue.

1 Like
7

USER:
How can I access the installation services for specific items?

gpt:
I’m sorry, I cannot provide a response as the query was in English and the requirement is to respond in the same language as the user’s query.

1 Like
8

Hi @hsnawn - I apologize but I do not have capacity to review the root causes for individual failed prompts.

If you could share observations on patterns when these issues arise, this will be much more helpful to evaluate what might be causing it.

9

I apologize but I can not see any patterns in this. It answers in random languages (Spanish, French, German) to queries in English. Sometimes this happens in the first query. Sometimes 20th query in conversation is answered in another language. It is so annoying.

10

I don’t know how involved you want to build this, but one way to absolutely cement this would be to use a trailing system message with something like this:

Reply to the user message following the initial instructions. Reply in this schema:

{
    "language": string, //ISO language code of the user's last message. Can be EN.
    "response": string // your response. Ensure it's in the same language as the previously identified code
}

Make sure your reply is JSON compliant following the schema above. Begin your reply with {

basically a micro-CoT.

But while this may work, I suspect you’re gonna have issues with other aspects of your prompt as well. :confused:

Edit: I didn’t realize you were talking about instruct, but you have a very conversational structure - why not use chat mode?

2 Likes
11

In addition to Diet’s point, I would like to clarify what you mean by “Query” in the context used above? I may be misinterpreting this and I am trying to rule out that the instruction “Answer must be in the same language as QUERY” contradicts the previous instructions and could thus contribute to the issue.

12

Sorry my bad, I am using the chat model, not the instruct. I have edited it.

13

You can also use something like What is language detection in Azure AI Language? - Azure AI services | Microsoft Learn

14

Welcome to the OpenAI dev community @hsnawn

I’d recommend simply mentioning “You reply in user’s language.” in the first system message and remove the “multi-lingual” part.

Additionally, remove the query from the system message and let it be added to the messages list as a separate message with user role.

Supply the context from previous conversation in a separate system message.

You can refer to this tutorial for the messages structure.

1 Like
15

Hi @hsnawn ,
Does this resolved ? I’m facing the same issue.