Hi @stevenjusher !
Short answer is that LLMs in general cannot approximate word counts well. This is something intrinsic to the LLMs, both due to how “words” are represented (as tokens, which may be sub-words, and may contain whitespaces), and also due to LLMs not really understanding the concept of words counts as a whole. You can tell them to be in depth, concise, long, or short, but then you have to build solutions over the top to take care of word counts.
There have been multiple such topics in the past, e.g. see:
- Can't get a model to follow a specific length / word count
- ChatGPT-4 Failing to deliver requested word counts and then stating incorrect word counts
- GPT Cannot Count Words - Why?
etc.