Understanding of logprob and top_logprob

[_j]

Sampling is used to randomly select from the set of tokens, biased by the certainty.

Selecting only the top choice is called “greedy sampling” and actually doesn’t produce language that seems as human, despite being more reliable.

How tokens are sampled can be affected by API parameters.

Questions regarding API sampling parameters (temperature, top_p)

Top-p reduces the selectable tokens from the 100% probability space at 1.00 down to only those that include the top 20% (inclusively) at 0.20. It is not affected by temperature. This will make probabilities like: “banana” : 30% “apple”: 20% “orange”: 15% (all others): 35% into a limited set of logits such as: “banana” : 66.66% “apple”: 33.34% Temperature then acts, for example lowering the temperature gives more distance between choices: “banana” : 75% “apple”: 25% (temperature work…

or

Is there a single sampling method used during inference, or there's a logic to use different sampling methods based on a given input?

The whole point is NOT to have deterministic or top token results. The use of unexpected words makes the response more passable as human-like. ChatGPT (the web chatbot) doesn’t allow you to access any parameters - diversity is also the point there, something for you to dislike if you don’t enjoy the production, something to upvote if it was favorable, to gather training data. The API of OpenAI presents two sampling parameters: top_p (nucleus sampling) which constrains the token dictionary, a…