Simple Request exeeds Completion Token Limit of 4096 Token; GPT-35-turbo

API
,
1

Hi Openai Community,

maybe you can help me. Am I overseeing an issue or something?

My request exeeds the completion token limit of 4096 with this simple request:

{"model":"gpt-3.5-turbo-1106","temperature":0.0,"messages":[{"role":"user","content":"Act as an professional translator to translate German into Hungarian we are an e-commerce motorbike business which sells spare parts and accessories. Translations must be performed in technical an e-commerce context. Be as accurate as possible, nouns must be capitalized. You receive a JSON object to translate. Do not change the object notation, keep objects and arrays as they are. No extra new lines, no new keys, values. Do not translate keys, only values. Translate only values of key-value pairs. This is my JSON object as string:{\"ARIDNR\":93933,\"ARBEN\":\"Tuning Vergaser 19N1-12 - für Simson KR51 Schwalbe, SR50, SR80\",\"ARKATERW\":\"Tuning Vergaser in Originaloptik für KR51/1, KR51/2\\n\\n- Replika-Produkt\\n- sieht aus wie der originale 16N1 Vergaser \\n- gute Qualität\\n- Aluminium-Guss\\n- kann bis zu 10% mehr Leistung bringen (in Verbindung mit weiteren Tuning- Maßnahmen)\\n- es empfiehlt sich auch ein Düsensatz zur Feinabstimmung!\\n- Ohne Benzinschlauch\\n\\nACHTUNG: Auf der Seite ist der Schriftzug \\\"19N1\\\" eingeprägt\",\"ARONLKEYWO\":\"Vergaßer Hauptvergaser\",\"Bullet1\":\"Replika-Produkt in Originaloptik\",\"Bullet2\":\"Gute Qualität aus Aluminium-Guss\",\"Bullet3\":\"Kann bis zu 10% mehr Leistung bringen\",\"Bullet4\":\"Empfohlen mit Düsensatz zur Feinabstimmung\",\"Bullet5\":\"Passend für Simson SR50, KR51/1 und KR51/2 Schwalbe\"}"}],"response_format":{"type":"json_object"}}

The Response looks like:

{
  "id": "chatcmpl-8NoJQsmRc1rmeW75fz3QCVzy6rtCG",
  "object": "chat.completion",
  "created": 1700685636,
  "model": "gpt-3.5-turbo-1106",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"ARIDNR\":93933,\"ARBEN\":\"Tuning Karburátor 19N1-12 - Simson KR51 Schwalbe, SR50, SR80-hoz\",\"ARKATERW\":\"Tuning Karburátor eredeti kinézetben KR51/1, KR51/2-hez\\n\\n- Replika termék\\n- úgy néz ki, mint az eredeti 16N1 karburátor\\n- jó minőség\\n- alumínium öntvény\\n- akár 10% -kal több teljesítményt hozhat (tuning intézkedésekkel kombinálva)\\n- Finomhangoláshoz ajánlott fúvókasor is!\\n- Benzincső nélkül\\n\\nFIGYELEM: Az oldalon a \"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    "
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 389,
    "completion_tokens": 4096,
    "total_tokens": 4485
  },
  "system_fingerprint": "fp_eeff13170a"
}

More information for context:

We translated nearly 20.000 products in 10 different languages using this prompt and comparable Json Objects took maybe 600-1000 completion tokens, so do you see where the problem could be?

We are unsing gpt-35-turbo 1106

1 Like
2

Important: when using JSON mode, you must also instruct the model to produce JSON yourself via a system or user message. Without this, the model may generate an unending stream of whitespace until the generation reaches the token limit, resulting in a long-running and seemingly “stuck” request. Also note that the message content may be partially cut off if finish_reason="length", which indicates the generation exceeded max_tokens or the conversation exceeded the max context length.

You may instructed it as so. No offense I’m not going to try and find it through the one line you posted.

This is why you are getting maxxed though (most likely) and should consider

A) Refining your instructions
B) Using a model with a higher token limit (if it’s not your instructions)

Looking at the response (Which comes out to ~200 tokens, not 4096), I’m guessing it’s:

Also note that the message content may be partially cut off if finish_reason="length", which indicates the generation exceeded max_tokens or the conversation exceeded the max context length.

1 Like
3

Thanks for your feedback. I tried using gpt-4-1106-preview and tada:

{
  "id": "chatcmpl-8NoqIiBdChpFC355EOj82ZqDvy6V0",
  "object": "chat.completion",
  "created": 1700687674,
  "model": "gpt-4-1106-preview",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\n  \"ARIDNR\": 93933,\n  \"ARBEN\": \"Tuning karburátor 19N1-12 - Simson KR51 Schwalbe, SR50, SR80 típusokhoz\",\n  \"ARKATERW\": \"Eredeti megjelenésű tuning karburátor KR51/1, KR51/2 típusokhoz\\n\\n- Replika termék\\n- úgy néz ki, mint az eredeti 16N1 karburátor\\n- jó minőség\\n- alumínium öntvény\\n- akár 10%-kal több teljesítményt nyújthat (további tuning intézkedésekkel együtt)\\n- ajánlott finomhangoláshoz fúvókakészlet\\n- benzincső nélkül\\n\\nFIGYELEM: Az oldalán a \\\"19N1\\\" felirat van bevésve\",\n  \"ARONLKEYWO\": \"Főkarburátor, karburátor\",\n  \"Bullet1\": \"Eredeti megjelenésű replika termék\",\n  \"Bullet2\": \"Jó minőségű alumínium öntvény\",\n  \"Bullet3\": \"Akár 10%-kal több teljesítményt nyújthat\",\n  \"Bullet4\": \"Finomhangoláshoz ajánlott fúvókakészlet\",\n  \"Bullet5\": \"Simson SR50, KR51/1 és KR51/2 Schwalbe típusokhoz illik\"\n}"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 349,
    "completion_tokens": 361,
    "total_tokens": 710
  },
  "system_fingerprint": "fp_a24b4d720c"
}

Completion Tokens is just 361 ?!

But why is that?

1 Like
4

Speculation but I’d imagine the newer/upgraded models are more reliable with the forced JSON mode.

It seems like the model you were using was entering an infinite loop until the token limit was hit and cut off

2 Likes