This is the prompt and response I got and the respective counts.
[Prompt]
Joker is a clown that tells a funny joke when asked to. If a topic is specified in the prompt, the joke needs to be relevant.
Q: Tell a joke.
Joker: What’s the best thing about Switzerland? I don’t know, but the flag is a big plus.
Q: Tell a joke.
Joker: Three men enter a bar in the USSR. One says, ‘Why did Stalin only write in lowercase?’ The other one says, ‘Because he was afraid of capitalism.’
Q: Tell a joke about games.
Joker: Yesterday I saw a guy spill all his Scrabble letters on the road. I asked him, ‘What’s the word on the street?’
Q: Tell a joke.
Joker:
[Response]
What do you call a woman who is working on her computer? A gamer.
Note: There is a single space character before the response.
Hi there, whitespace and new lines also add tokens (which you may have stripped?). I see that in your screenshot there are also values after the written completion ("| 183…").
“| 183” is not part of the response and it has been added for debugging purposes.
I repeated the measurements again with 4 different responses and the same prompt as in the original post. The bold text is not part of the response and shows my calculated prompt + response tokens.
I don’t know, but I think it’s a good thing.(167 + 14)
This joke is so dirty that even my mom would blush.(167 + 12)
What did one monkey say to another after a close call?(167 + 12)
I went to the supermarket to buy my wife a present. I saw her favourite brand of beer, so I bought it for her. But she wouldn’t accept it, saying that we don’t drink beer anymore, and that there’s no point in buying something which will soon be out of date. I asked her why we stopped drinking beer, and she said that after a week of not drinking any alcohol at all she was going crazy with withdrawal symptoms.(167 + 91)
Note: There is a single space character before every response
Hi there, would it be possible to share a playground link so I could try to replicate this behaviour? As now I’m just seeing pasted completions, it’d be great to see the behaviour first-hand.
First of all why we even need to calculate tokens using like GPT2 tokenizer if out dashboard knows how much token we used why GPT3 API just send same as in response body please @joey let me know if I am wrong here because that’s really important when we make our in productions
Following up on this - barring progress on the token count being included in the API, could you confirm that HF’s GPT2Tokenizer.from_pretrained(“gpt2”) is the one we should be using (versus something from GPT2TokenizerFast)?
