Empty content following rightful response to use up all max_token

API
1

this happens 1 in 10 calls

normally my response content looks like

    {
        "name": "some data",
        "tactic": "some data",
        "content": "some data"
    }

and it counts around 500 tokens, which is just what I expect from the api, but sometimes it returns

    {
        "name": "some data",
        "tactic": "some data",
        "content": "some data"
    }

it added a loooooot empty space or newline and the completion token showed that this response used all of my max_token that I set.

using gpt-4-1106-preview, below is the code making calls to the model

response = openai.chat.completions.create(
        model=input_model,
        response_format={ "type": "json_object" },
        messages=[
            {"role": "system", "content": "你是一个金牌顾问式销售，帮助构思1000字的话术，不要吹嘘自己的产品，语气不要太活泼也不能太官方，以客户朋友的口吻简洁有效的表达，用话术策略中的数据论证你的观点，不要引用不在策略内的数据，欺骗客户是很严重的问题。You always reply in JSON format"},
            {"role": "user", "content": prompt}
        ],
        max_tokens=800,
        seed=input_seed,
        n=1,
        stop=None,
        temperature=0.2,
    )
2

Could you just check whether this is happening for the 0613 model as well or only the 1106 model ?

Alternatively, you could get it to generate the json/dict within backtics and use those as stop tokens to end generation. That should take care of the excessive lines at the end.

Another possible case might be that if you are using samples, one of them might have some extra newlines in them

3

thanks for helping, did some testing.

seems like if you specify the response_formant in the request, content with not be surrounded by ```json and ```, removing it from the request and using \n``` as the stop token fixed it.

0613 seemed to be ok, tried a few times

1 Like