Currently, it’s only possible to encode tokens to their corresponding tokenID using the Tokenizer, however, I am wondering if there is a method to decode tokenIDs back to text?
Related topics
Topic | Replies | Views | Activity | |
---|---|---|---|---|
Is there a way to make a tokenizer using tiktoken lib | 0 | 172 | September 21, 2024 | |
Using a Custom Tokenizer with GPT Embeddings | 5 | 3815 | March 4, 2024 | |
Token number inconsistent between API and ChatGPT | 1 | 770 | August 23, 2023 | |
Feature request: get generated tokens back with request | 5 | 897 | December 15, 2023 | |
Converting embedding vector to text | 16 | 19056 | July 24, 2024 |