Should we finetune only in English?

[3WaD]

Hello. Today I successfully tested my first chatGPT3.5 finetuning based on the new documentation. The conversation data I used was not in english, but czech. Since that’s the language we use to speak with the model in our final usecase. But the result seems to just break the model’s responses completely, making it unusable rather than any better. So my question is, since I saw no mention of it anywhere, should we translate our data into english for good results first?

[_j]

I don’t see any definitive reason why any language can’t be used as training. The same language as all input I would expect to strengthen the reinforcement of responses.

Consider if other recommendations are being used:

• The system prompt should be included in fine-tuning, should be the same as you are using in your application, and should give the AI a clear new identity to ensure it already recognizes the fine-tuned purpose even before you talk to it.
  Example: “Jste AnswerBot, užitečný AI asistent vyškolený pro Českou republiku Turistickou agenturu.”
  (translation: You are AnswerBot, a helpful AI assistant trained for the Czech Republic Tourism Agency.)

• An example fine tune should have a sequence [system/user/assistant]. Others are supported, but how they may behave is not documented.

The system prompt being in Czech may have one difference, given that we know that “system” lets us decide on a new fine-tune identity: existing chat tuning will be dominated by “You are a helpful assistant” or “You are ChatGPT” (when trained including a system prompt). Your new system message might de-emphasize prior training even more than if still in English, even when it is “how to chat”.

[3WaD]

Ok after a while I managed to improve the results drastically. The key is the Temperature value. While the non-fine-tuned models are doing very well even at default 1, my fine-tuned model has to have this value waay lower to be usable and to actually feel the effect. Maybe obvious for more experienced users, but seems like I am not . So if anyone having weird results from fine-tunes, try playing with that before doing anything else.

[_j]

Yes, that is good information I shared before, but didn’t think of here because of your specific use.

Grammar errors, misspellings and errors of masculine and feminine

The temperature setting that ChatGPT uses internally may be too high for the perplexity of less common languages in the training data. You could reinforce Urdu first by pasting some large texts in the tone of the language you want it to understand. Make a category header like “// language style and usage training //” in Urdu so AI knows it is not directly something to answer about. The API, and OpenAI’s platform.openai.com playground may be a place for testing the models directly to see if low…