Character encoding (black diamond output)

General API discussion
1

Does anyone know how to make OpenAI output in the proper encoding? I get the black diamonds in a lot of my non-English outputs and I can’t figure out why.

For example,
platnosnos��

What I’ve tried:

  1. Adding charset=UTF-8 in the CURL header. Content-Type: application/json; charset=UTF-8

  2. Adding the proper encoding to the prompt(s), i.e. at the end “Character encoding: X”.

I check console and the outputs with black diamonds are coming directly from OpenAI (i.e. it’s not my website/database that’s the problem, I have the proper encoding in the META).

Has anyone figured out a way to solve this?

2

The models aren’t primarily trained on other languages, but they do okay occasionally.

I’ve not seen black diamonds for missing, though. What system are you on? What’s the prompt look like? Maybe OpenAI is sending right character’s and you can’t seen them on your system?

3

I’d say the models perform quite well in other languages, not just occasionally. The translations/outputs are almost as good, if not as good, as Google translate (using davinci-003).

I’m pulling the black diamonds directly from console, so this is what I’m receiving directly from OpenAI.

For example:
je dôle��ité nájsť takého poskytovateĺa

You can tell it to write anything in any language and eventually, you’ll get this type of response.

1 Like
4

What’s your temp and frequency_penalty? Which model?

5

Davinci-003
Temp: 0.8
FP/PP: 0.5