Is it possible to get the entire logits/softmax probability vector of the output?

Is it possible to get the entire logits/softmax probability vector of the output by interacting with the API? If not then how much can we get?

You probably know that returns are limited to 5 logprobs. Previously the count was higher, I think 8. You get it for every token. Only supported on completion endpoint models. It would be interesting research to get a much longer list.

math.exp(log_prob) to get standard probability.

It’s not really a vector, but a per-token probability.

You can compute your own scoring of a result on logprob output, or even compare that total score of generation to the top per-position. Each n + 1 relies on the language previously generated, so there is also a “best path” of generation at top_p = 0.0.

You can highlight a run of tokens in the playground and get a total score, just an addition that tells you an entire sequence was very unlikely, apparently, and gets more unlikely the longer it is.

The best-of formula for API to return the “best” of several trials doesn’t seem to be disclosed.

I just want to check if they have made any changes recently. Sad to know it is still very limited for my use case. Guess I have to resort to open-source model then…

If you are just looking for more logprobs beyond the five, Azure has this written for their completion documentation:

For example, if logprobs is 10, the API will return a list of the 10 most likely tokens

So maybe they could give more if true.

FWIW, they plan on bringing logprobs to chat endpoints too:

1 Like