Logit Bias default bias and blocking tokens not in list

API
1

How do I set a default logit bias value. Say I want to create generations where only certain tokens are allowed and to block all other tokens from showing up, how would I achieve this? I could possibly get all tokens and unless they are in my list, set their bias to -100 however I’m not sure how to get a set of all the tokens in the vocabulary and I’m also wondering if there’s a better way.

2

You cannot demote every of 200000 logits.

Invalid 'logit_bias': too many properties. Expected an object with at most 1024 properties, but got an object with 10000 properties instead.

Instead, you would have to highly promote only those you want.

Additionally, this will prevent the AI from ending the output, a token you cannot control. You would have to use a very short max_response_tokens parameter.
Otherwise, provide a stop sequence and a token of finishing that you still can’t trust.

Then finally, validate across the input cases to see if the reproduction and determinations are still as you expect. If you only want the words “good” “neutral” “bad”, you have to ensure they still have the original qualities and ratios, which you can examine by logprobs.

3

If I wanted “good”, “neutral”, “bad” I would probably do some prompt engineering paired with a structured output. What I want is more like only allowing “uncomplicated” words, actually my task is for language learning. I want to only allow tokens the user has an understanding of, which would be in the range of a few hundred at the minimum. And I need it to be more sophisticated, for example I wouldn’t fully block tokens, but simply demote them so that if the model strongly predicts that token it would have no choice but to select it even if the user doesn’t know what it means. Also I would of course not include special characters in the demotion, so the stop token wouldn’t be an issue.

4

Welcome @ubersuperrareboss

Logits are not the best way to achieve what you have described, in my opinion.

Here are two much more convenient and effective methods:

  1. Pass the model the vocabulary to use within the system message along with an appropriate instruction.
  2. You can also use grade level or age level descriptions within the system message. e.g., Please make sure that your vocabulary is lucid for a 2nd-grade student.

A good prompt would be able to influence the logprobs to favor your expected output.