1

I’m using logit bias quite a bit to adjust generated tokens, and I’d like to do something similar with signs. For example - to increase or decrease to probability of “new line” in the text. Is it possible?
Tried checking the tokenizer site, but it seems that ‘\n’ is divided into two tokens, so can’t really use that.

Thanks!

2

I am also trying to figure this out. Did you determine what it was or how to resolve the two tokens it was split into?

3

This will be a futile task. There’s thousands of tokens that end with newline.

Untitled

4

So fine tuning or prompting are the only ways to prevent it.
Thanks