Hello, I’m using OpenAI models for Arabic reading comprehension tasks, but I encountered a problem when using the cl100k tokenizer; the models didn’t reply to any of my queries; meanwhile, when I use the gpt2 tokenizer, it works as expected. Does anyone know why this may happen with Arabic texts?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| What cause the rubbish data output from GPT4 API calls response in arabic? | 4 | 823 | June 2, 2024 | |
| Right to Left languages token count | 1 | 935 | March 18, 2023 | |
| Gpt-4o-mini responses are being cut off | 1 | 202 | January 28, 2025 | |
| Struggling to get correct token count | 2 | 1893 | September 4, 2023 | |
| GPT-4 API Outputs Gibberish When Prompted in Arabic | 0 | 446 | May 7, 2024 |