I’m trying to develop a simple text completion app with GPT-3 for both English and Spanish. Although GPT-3 works perfectly in English, it sometimes answers in English to a prompt given in Spanish.
For example, with the model text-davinci-002, with temperature=0.0, if given the prompt “Hola, ¿cómo estás?”, it answers with “I’m doing well, thank you. How are you?”.
Thus, I would like to know if there is some kind of prompt engineering to make the model always answer in the input language (no matter what the prompt is) or maybe I should use Google Translate to translate from Spanish to English, submit the prompt in English and then translate the answer from English back to Spanish.
Sometimes adding “answer in Spanish” helps but in other cases it doesn’t. A few examples:
Prompt: Answer in Spanish. Hola, ¿cómo estás? Completion: Hi, how are you?. ¿Cómo estás?. \n I’m good, thank you. ¡Estoy bien, gracias!.
Prompt: Answer in Spanish. Escribe una historia. Completion: Once upon a time, there was a beautiful princess who lived in a faraway kingdom.
As you can see, it is still answering in English to spanish prompts. When I change Answer in Spanish for Responde en español, it sometimes works and other times doesn’t. For example, it now answers the second example in Spanish but the first one still in English.
By the way, I’m using the text-davinci-002 engine (the most powerful one) with temperature=0.
Why do you have the temperature set to 0? That’s likely your problem. The vast majority of what GPT3 was trained on was english, therefore english tokens are the most likely response. Make sure you set P to 1.0 as well.
I have now tried different temperature values and, as you said, it helps. With a temperature of 0.8 and adding the text Responde en español before each prompt, it now answers in Spanish most of the time.
However, in approximately 10% of the cases, it still introduces English text.
Example: Prompt: Responde en español. Escribe una historia. Completion: Mi abuela siempre me cuenta historias interesantes. One day, my grandmother told me a story about a young woman who lived in a small village.
I am unsure whether this is the best approach to produce non-english text or maybe I should simply query GPT-3 in English and use Google Translate (or any other service) to translate from English to the desired output language.
With an even higher temperature of 0.95 it works even better. However, it seems that no matter how high I set the temperature, it still occasionally answers in English to Spanish.
You might try a few-shot approach then, or go with the original model. I suspect the INSTRUCT series models were mostly trained on english instructions.
The issue with the original, i.e., non-instruct, models is that they usually require more complicated prompt engineering.
For now, I think I’ll stick to instruct-GPT models with large temperature values and adding “Responde en Español” to each spanish prompt, since in most cases the model actually answers in spanish. An occasional answer in the incorrect language is not a really big issue since I’m using GPT-3 with educational purposes.
In my experience the insertion model is easier to prompt engineer and works better than the other non-instruct models, since it’s less likely to think it has completed its task and want to start something new.
My app will be used by users with no knowledge whatsoever about prompt engineering, so they must be able to give orders such as “Write a story” or “Tell me five cool names for dogs”, just as it is possible with the instruct models.
If I have understood you correctly, @TimC, you are suggesting using the non-instruct models in Edit mode. Does this use case require any type of prompt engineering or I can use them just as the instruct models (i.e., give them orders)?
