Is it possible to get the entire logits/softmax probability vector of the output by interacting with the API? If not then how much can we get?
You probably know that returns are limited to 5 logprobs. Previously the count was higher, I think 8. You get it for every token. Only supported on completion endpoint models. It would be interesting research to get a much longer list.
math.exp(log_prob) to get standard probability.
It’s not really a vector, but a per-token probability.
You can compute your own scoring of a result on logprob output, or even compare that total score of generation to the top per-position. Each n + 1 relies on the language previously generated, so there is also a “best path” of generation at top_p = 0.0.
You can highlight a run of tokens in the playground and get a total score, just an addition that tells you an entire sequence was very unlikely, apparently, and gets more unlikely the longer it is.
The best-of formula for API to return the “best” of several trials doesn’t seem to be disclosed.
I just want to check if they have made any changes recently. Sad to know it is still very limited for my use case. Guess I have to resort to open-source model then…
If you are just looking for more logprobs beyond the five, Azure has this written for their completion documentation:
For example, if logprobs is 10, the API will return a list of the 10 most likely tokens
So maybe they could give more if true.
