What is the reason for adding total 7 tokens?

What, exactly, are you trying to ask?

<|im_start|>user<|im_sep|><|im_end|>

Is four tokens: 100264, 882, 100266, 100265.

<|start|>assistant<|im_sep|>

Is three tokens: 100264, 78191, 100266.

1 Like