ChatGPT stops generating mid-translation

API
1

Hi, I am trying to do translation using GPT3.5-1106 and running into an issue where for some input strings the model will randomly stop generating before the end of translation. The issue can be reproduced in the following playground - https://platform.openai.com/playground/p/KbVsWFeehaXr66DepzdXKyl5?mode=chat. Note that sometimes the model generates the correct output but roughly 3/10 times it stops in the middle of generation.

The same input seems to be working with GPT3.5-0613.

2

Hi, and welcome! I understand you’re curious about why a certain AI model stops writing a translation. It’s a great question and I’m here to help clarify this for you.

AI models, like the one you’re using, are designed to balance efficiency and accuracy. However, with lower costs and faster output, there can be a higher degree of uncertainty, which in AI terms, we refer to as ‘perplexity’.

The playground preset you shared operates based on default sampling parameters, namely ‘temperature’ and ‘top_p’. These parameters influence the model’s decision-making process. For instance, if the AI predicts it’s time to emit an end-of-text stop token with only 5% certainty at the end of any sentence, it will indeed stop 5% of the time.

In the context of less-certain world languages, it’s particularly important to reduce both these parameters. This can help in improving the model’s performance and reducing abrupt stops.

To understand the model’s performance better, you can measure how much the AI actually wrote. Typically, you’ll find the AI wrapping up things around 1000 tokens or less, regardless of the input.

For your specific use case, I recommend using the gpt-3.5-turbo-0125 model. It’s a suitable replacement for the 1106 model. Remember, the key is to find the model that works best for your specific needs, even if it means trying out a few before settling on one.

I hope this helps! If you have any more questions or need further clarification, feel free to ask. Happy experimenting with AI! I even made this message more “welcoming” with AI!

3

Hi, the suggested solutions do not resolve the issue. You can verify this with the playground link I shared.

4

I can fix the playground link by not having it look like the AI has already provided an answer.

System:

You are TranslateBot.

User:

// Instruction to AI

Translate the following English text to Vietnamese:

// end of Instruction to AI

// text to translate

Overall, your language skills are effective in communicating your ideas, but there are some areas for improvement:

  1. Relevant and engaging responses:
  • You provided brief answers to your friend’s questions, which limited the conversation and made it feel less engaging.
  • It would have been helpful to provide more details about your preferences for the date night and ask your friend for their ideas as well.
  1. Clear and fluent language:
  • Your language was generally clear and understandable, but you had some difficulty finding the right words and constructing complete sentences.
  • It would be beneficial to expand your vocabulary and practice using more complex sentence structures.
  1. Using the right tone:
  • Your tone was polite and friendly, which created a positive interaction with your friend.
  • However, you could work on expressing yourself with more confidence to demonstrate your language proficiency and engage your conversation partner more fully.

// end of text to translate

5

Of course, you can use system messages, but the problem still occurs even with system messages added.

This problem might be related to the type of tokenizer used in gpt-3.5-turbo-1106.
How about trying it with max_tokens=3000?

I suggest this because I’ve noticed that the finish_reason sometimes becomes “length” when the translation result gets cut off in the middle after several attempts.

6

Please refer to the screenshot for your reference.

image

7

I accounted the same issue and the reason was because of the safety measure.

The document was a legal agreement and I tried to translate it into another language. I found out GPT3.5/4.0 Claude all stopped at the same position.
Then I carefully read the part and I found that a word ‘termination’ is mentioned a lot.

Not found a solution yet.

1 Like