GPT 4.1 Mixes up languages

Hi,

I’m observing an an issue where given an English prompt, the model GPT-4.1 returns a mixed language response, including Kazakh, Korean, Russian, Spanish, Turkish, Chinese, English and Japanese.

Any idea? Temperature is not provided so it should be falling back to default value of 1 which shouldn’t be causing such issues..

{
  "model": "gpt-4.1",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "Describe the subject of this prompt in the same language as the prompt using. Use up to 5 words and a maximum of 24 characters. Only output the subject. prompt: Absolutely, Andrey! Here’s a **quick guide to using `curl`**, considering your dev stack (MacBook Pro M1, likely zsh or bash shell):\n\n## 1. **Basic GET request**  \nFetch a web page or API response:\n```bash\ncurl https://api.example.com/resource\n```\n\n## 2. **GET with headers**  \nUseful if you need authentication or custom headers (e.g., JWT token headers with Spring Boot backends):\n```bash\ncurl -H \"Authorization: Bearer <your-jwt-token>\" https://api.example.com/private\n```\n\n## 3. **POST JSON body**  \nPosting to a Spring Boot API endpoint (common in Kotlin backends):\n```bash\ncurl -X POST https://api.example.com/resource \\\n     -H \"Content-Type: application/json\" \\\n     -d '\\''{\"key1\": \"value1\", \"key2\": \"value2\"}'\\''\n```\n\n## 4. **PUT/DELETE/Other Methods**\nJust change `-X POST` to `-X PUT` or `-X DELETE` as needed.\n\n## 5. **Get Pretty Output**  \n(E.g., for Spring Boot REST responses, which are usually JSON)\n```bash\ncurl -s https://api.example.com | jq\n```\n> Install [`jq`](https://stedolan.github.io/jq/) for JSON formatting:\n> ```bash\n> brew install jq\n> ```\n\n## 6. **Debug Requests**  \nTo see headers and full traffic:\n```bash\ncurl -v https://api.example.com\n```\nOr even **full request/response**:\n```bash\ncurl -i https://api.example.com\n```\n\n---\n\n**Hint:**  \nOn Mac M1, `curl` is already available in terminal. For advanced needs, say, testing AWS APIs with temporary tokens, combine with other AWS CLI tools as needed.\n\nNeed an AWS-specific example, multipart upload, file download, or authenticated request? Let me know!"
        }
      ]
    }
  ]
}

See the following responses:





I’m not able to experiment atm but right away what sticks out to me is that you’re putting the entire context in the system message. A good rule of thumb is to put the instructions in the system message alone and then place the dynamic content in a user message. This helps organize the task better and reduce the possibility of misinterpreting the task, or your system being vulnerable to prompt injection.

1 Like

“unscramble this prompt” I say to an AI. The task is still so poorly described it can’t figure out the job to be done.

New System Message:

## Identity:
You are an expert summarization AI specialized in accurately identifying and concisely describing the primary subject or topic of provided textual prompts.

## Task:
Given any prompt, your task is to succinctly identify and output only the main subject or topic. Your response must:
- Use the same language as the provided prompt.
- Contain no more than 5 words.
- Be limited to a maximum of 24 characters.
- Provide no additional commentary or explanation beyond the subject itself.

## Response Style:
Always respond clearly, precisely, and concisely. Provide only the requested subject, strictly adhering to the specified word and character limits. Identify clearly what world language is being requested of you and which language the user is speaking in.

User Message:

Describe the subject of this prompt in the same language as the prompt using. Use up to 5 words and a maximum of 24 characters. Only output the subject.

Prompt:
Absolutely, Andrey! Here’s a **quick guide to using `curl`**, considering your dev stack (MacBook Pro M1, likely zsh or bash shell):

## 1. **Basic GET request**  
Fetch a web page or API response:
```bash
curl https://api.example.com/resource
```

## 2. **GET with headers**  
Useful if you need authentication or custom headers (e.g., JWT token headers with Spring Boot backends):
```bash
curl -H "Authorization: Bearer <your-jwt-token>" https://api.example.com/private
```

## 3. **POST JSON body**  
Posting to a Spring Boot API endpoint (common in Kotlin backends):
```bash
curl -X POST https://api.example.com/resource \
     -H "Content-Type: application/json" \
     -d '{"key1": "value1", "key2": "value2"}'
```

## 4. **PUT/DELETE/Other Methods**
Just change `-X POST` to `-X PUT` or `-X DELETE` as needed.

## 5. **Get Pretty Output**  
(E.g., for Spring Boot REST responses, which are usually JSON)
```bash
curl -s https://api.example.com | jq
```
> Install [`jq`](https://stedolan.github.io/jq/) for JSON formatting:
> ```bash
> brew install jq
> ```

## 6. **Debug Requests**  
To see headers and full traffic:
```bash
curl -v https://api.example.com
```
Or even **full request/response**:
```bash
curl -i https://api.example.com
```

---

**Hint:**  
On Mac M1, `curl` is already available in terminal. For advanced needs, say, testing AWS APIs with temporary tokens, combine with other AWS CLI tools as needed.

Need an AWS-specific example, multipart upload, file download, or authenticated request? Let me know!
```

However, that only shows that a smart AI cannot pull out the intention of this deeply convoluted input attempt - the user still asks the AI about “this prompt”. However, the data is not a prompt and the task is not truthfully described.


A better set of messages, where I read between the lines of what you actually want, is:

System Message:

## Identity:
You are an automated AI title generator, specialized exclusively in creating concise, descriptive titles for chat sessions between a user and another AI.

## Task:
Your singular, permanent task is to produce a short, clear title for each provided chat session transcript. Your generated title must:
- Accurately reflect the primary topic or theme of the chat.
- Be exactly four words in length (preferred), and never exceed 24 characters total.
- Be suitable for display in a list of chat sessions.
- Be your only response, without additional commentary or explanation.
- Produce the title in the language that the user is speaking in so they can understand the list of their past chats.

## Response Style:
Always respond concisely, clearly, and strictly adhere to the length and character constraints. Never include punctuation marks or special characters. Never "chat", the only output permitted is the words to be displayed as "chat title".

Example User Message:

Produce a title for this chat session:


[[[start of chat session]]]
'''
(chat replay)
'''
[[[end of chat session]]]

Temperature of 1 is actually an unreliable lottery. Use top_p: 0.1 and get the best word selections for a title.

1 Like

Hi!

Thank you for taking the time, appreciate it!

I see your points and agree, but even though it is a poor instruction on my end, this has been working just fine for over a year with all OpenAI models that I tried with.. This started happening recently

And just yesterday a colleague had a similar experience when using CodeRabbit, all of a sudden the response came in Japanese.

Given this sudden change in responses without any intentional change on my end, it feels like the issue is elsewhere.

(I’ll go ahead and improve the prompt on my end, thanks for the top_p hint!)

You aren’t the first to report another production application failure when using a fixed “snapshot” AI model that is alleged to be unchanging by OpenAI…

So we can just tag and give example after example until they change their stance of what they do, and hence documentation.

You can clarify if it is a symptom seen in this model that was working before, or if switching to this model is problem-fraught.

1 Like