You can use the tiktoken library, locally, to encode strings and decode token number sequences. Doing so, you could see the token numbers, delineation between tokens, and obtain counts.
The embeddings still accepts just plain text, and the token operation behind the scenes really isn’t of much application, unless, for example, you wanted to run embeddings on every of 100k tokens to see how similar "psy"
is to " Psycho"
. (the former is if you input psycho lower-case on a new line.
Embeddings is much more concept-based than words. It will have an overview of this entire post when sent, comparable against other text in immeasurable ways.