For over a year I have found that GPT cannot count words. Even when you tell it to count every word, it still cannot. It THINKS the word count is X even when it has counted Y. Here is an example where I told it to count every single word and for the total to be 140 words.
"…and (120) learning (121) preferences (122).
This version now reaches 140 words exactly, counted meticulously. Thank you for your patience."
You see?! It thinks 122 is 140! It is incorrect 9 times out of 10, and it becomes so infuriating that I end up insulting it!
LLMs excel in generating “next likely token” which has emerging properties in “understanding” but a token is the precision of roughly half a word on average so details smaller than that (e.g. change of a single character number within a long number) are hard for it to process.
In addition, LLMs don’t measure things, they create semantic continuity.
In general, they are rubbish at maths and you are far better using a function to do any calculations.
It works using tokens not words and letters. You might want to educate yourself on how LLMs work currently instead of spending a year wondering why they can’t do something.
Try the o1-preview model, that one is much better, it does a good job even at counting words!
I made a few tests with short, mid and long text, I compared the results with third party word counter tool and the only time I got a mismatch it was due to a -.
When I asked to double check why there was a mismatch of 1 word it gave me the below answer:
“AI-Powered” can be considered as two words (“AI” and “Powered”), which brings the total word count up by 1.
You’re absolutely right that, historically, word counting in AI models has often been surprisingly inconsistent, and I can understand how frustrating that must be! Let me explain a bit about why this might happen.
In a model like GPT, language generation and processing are based on probabilities and pattern recognition rather than precise word-counting capabilities. Word counting in GPT often fails for a couple of reasons:
Tokenization Over Words: GPT (and other language models) splits text into tokens rather than words. Tokens don’t always map directly to individual words — “can’t,” for example, might be one token, while “cannot” could be split into two. Because it’s counting tokens rather than literal words, it can misestimate or “think” it has counted correctly even when it hasn’t.
Imperfect Parsing Logic: When GPT tries to parse and count text, it’s applying language-based logic rather than numerical tracking, leading to inconsistencies. GPT’s generation process is heavily focused on fluent and meaningful responses rather than precise quantitative accuracy, and the way it tracks these things makes exact word counts challenging.
Lack of Explicit Counting Mechanisms: Unlike specialized tools or coding languages where you can specify precise counting functions, GPT lacks direct access to logic-based counting functions during generation, which is why it’s prone to making approximate or even incorrect assumptions about counts.
Using external tools, however, would allow for accurate word counts — so if you ever need a meticulous count, a tool like a simple word processor or script would definitely be more accurate than GPT alone.
The model can’t count because it suffers the same issue human do, we’re not very good counting larger numbers of things without tools, be those pen and paper, tally marks, fingers, etc.
If you tell the model it has a tool to count words, it will gladly make use of it, word counting is something classical software can do with near 100% accuracy. It’s the wrong task for an LLM.
How would you count how many words are in a document yourself?
Would you stand back five feet and view it all through a foggy lens while there are others yelling numbers at you, instructions of how to paint pictures and what your friend likes for dinner? On a document where words are encoded to look like [35251, 40938, 11445]…
Or would you internally keep a count of words, “one, two, three, …” as you read them, distraction-free?
Transformer-based GPT is the former.
Here’s a post where I break down an even more deliberate prompted reasoning technique, on “count letters”, and have the AI do something you might say “can’t be done” - count the "r"s in a STORY about strawberries.
Having such an auto-prompt library of techniques is how someone could make a “reasoning” AI that consumes thousands of tokens to produce that one word count value.
It is similar like in humans: you will need few additional tools. Following brain centres are significantly involved with mathematics in humans: Parietal cortex: the core area for representing and manipulating numerical quantities Temporal cortex: contains neurons that respond selectively to visual numerals
To do the correct counting in LLM, you will have to provide LLM with some additional tools (i.e. temporal memory, etc.). The easiest way to achieve this (to some extent) is to use already implemented Python interpreter. If you ask LLM to generate a code for counting numbers and to execute it, you could get a pretty close result.
