Arabic Texts Tokenization

Hello, I’m using OpenAI models for Arabic reading comprehension tasks, but I encountered a problem when using the cl100k tokenizer; the models didn’t reply to any of my queries; meanwhile, when I use the gpt2 tokenizer, it works as expected. Does anyone know why this may happen with Arabic texts?

1 Like