Currently, it’s only possible to encode tokens to their corresponding tokenID using the Tokenizer, however, I am wondering if there is a method to decode tokenIDs back to text?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Is there a way to make a tokenizer using tiktoken lib | 0 | 172 | September 21, 2024 | |
| Using a Custom Tokenizer with GPT Embeddings | 5 | 3815 | March 4, 2024 | |
| Token number inconsistent between API and ChatGPT | 1 | 770 | August 23, 2023 | |
| Feature request: get generated tokens back with request | 5 | 897 | December 15, 2023 | |
| Converting embedding vector to text | 16 | 19056 | July 24, 2024 |