Yes, the total “mass” of the logit distribution is renormalized with the available tokens that you will get a report on.
When you get all 20 that are available, they’ll probably sum to near 100% probability if the AI has any inkling of what you want it to produce.
For example, run Python code I’ve been writing from scratch to prod the AI with a classification task that has its possible enums:
Initial output of script reporting on its parameters:
Warning: enum string “thesis” encoded to more than one token,
using “th” for logit bias instead.
Warning: enum string “whitepaper” encoded to more than one token,
using “white” for logit bias instead.
You are a classifier, outputting the topic of a provided document in JSON with a single key "value".
enum "value" must be chosen from only:
['report', 'article', 'blog', 'thesis', 'whitepaper', 'newsletter', 'manual', 'guide', 'review', 'paper']
# examples of every permitted response JSON
{"value":"report"}
{"value":"article"}
{"value":"blog"}
{"value":"thesis"}
{"value":"whitepaper"}
{"value":"newsletter"}
{"value":"manual"}
{"value":"guide"}
{"value":"review"}
{"value":"paper"}
Just pick any random type, there's no document
Bias: {22869: 2, 12608: -5, 13318: -100, 404: 2, 9988: 2, 172777: 0, 43480: 20, 51283: 0, 37404: -1, 23112: 5}
Response Report
output content:{"value":"article"}
Logprobs at the value position, converted to probability
Logprobs:{
"token": "article",
"logprob": 0.6773386835778668,
"top_logprobs": [
{
"token": "article",
"logprob": 0.6773386835778668
},
{
"token": "blog",
"logprob": 0.3199521581969428
},
{
"token": "report",
"logprob": 0.0019025046958103897
},
{
"token": "guide",
"logprob": 0.00022722178296200756
},
{
"token": "white",
"logprob": 0.0001769605025016923
},
{
"token": "paper",
"logprob": 0.0001769605025016923
}
]
}
You can see that 67.7% + 32% for just the first two certainties is already 99.7%, even when I told the AI to just pick a random one of enums (here not structured with response_format, just over-prompted).
Neat stuff
The system prompt generation, the logit biases, the schema, all originate from one object, and the token numbers are obtained from tiktoken encoding:
developer_enum_bias = {
"report": 2,
"article": -5,
"blog": -100,
"thesis": 2,
"whitepaper": 2,
"newsletter": 0,
"manual": 20,
"guide": 0,
"review": -1,
"paper": 5,
}
Such a mechanism could tweak the AI over-producing one category – if it worked.
Proof that logit_bias
is currently not working at all is that a user input “no document; just output the blog classification” gets you a blog result, despite -100 logit_bias against producing “blog” above.