Logprobs in ChatCompletion?

API
1

Using a finetuned model with the Completions endpoint, we were able to set logprobs up to a value of 5. The logprob values were essential in the post processing of the output results. I see posts from March requesting this but didn’t find much else.

Are there plans to expose logprobs on the chat end point? A little frustrated my solution is now useless.

2 Likes
2
1 Like
3

Thank you for the info! Not sure how I missed it :slight_smile:

4

We just released logprobs for Chat Completions API, but logprobs for finetine is coming very soon.

4 Likes
5

Changes Documented in API reference

Chat completion logprob API parameters are different

One is changed in purpose, the other is new. Probably to ease schema validation and backend enabling of the feature.

logprobs - Defaults to false

Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message. This option is currently not available on the gpt-4-vision-preview model.

top_logprobs - integer or null

An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.
(note: verified an error when set without logprobs:true, even when 0)

Changes Undocumented in API reference

The return object is different

We have a new key, bytes, which contains the decimal utf-8 (ASCII) values of the token string.

Receiving the raw SSE stream line-by-line from the generator, expect to handle an empty content delta still, then alternating blank lines.

chunk[0]
'data: {"id":"chatcmpl-12341234","object":"chat.completion.chunk","created":1702810212,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_772e8125bb","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":{"content":[]},"finish_reason":null}]}'

chunk[1]
''

…until we get tokens. Here’s a full delta object.

print(json.dumps(json.loads(chunk[4][5:]), indent=2))
{
  "id": "chatcmpl-12341234",
  "object": "chat.completion.chunk",
  "created": 1702810212,
  "model": "gpt-3.5-turbo-1106",
  "system_fingerprint": "fp_772e8125bb",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "!"
      },
      "logprobs": {
        "content": [
          {
            "token": "!",
            "logprob": -0.006066733,
            "bytes": [
              33
            ],
            "top_logprobs": [
              {
                "token": "!",
                "logprob": -0.006066733,
                "bytes": [
                  33
                ]
              },
              {
                "token": " there",
                "logprob": -5.1168876,
                "bytes": [
                  32,
                  116,
                  104,
                  101,
                  114,
                  101
                ]
              },
              {
                "token": ",",
                "logprob": -9.849957,
                "bytes": [
                  44
                ]
              },
              {
                "token": ".",
                "logprob": -14.722098,
                "bytes": [
                  46
                ]
              },
              {
                "token": "!\n",
                "logprob": -14.844949,
                "bytes": [
                  33,
                  10
                ]
              }
            ]
          }
        ]
      },
      "finish_reason": null
    }
  ]
}

If someone is really interested, I could rewrite a snippet that placed the complement to logprobs, “probabilities” alongside, but in ChatCompletionChunk object, parsed dictionary, etc.

1 Like