ChatGPT cannot count words or produce word-count-limited text

[ruby_coder]

@Ozpeter

I guess I’m expecting too much too soon.

I think you and many others are looking at a beta marketing application called “ChatGPT” and you are trying to take this beta marketing application and program it like it was an API.

The API is called OpenAI API and requires a lot of programming to begin to “behave” like the ChatGPT web application.

Because of high demand for ChatGPT, OpenAI (and Microsoft) have already announced a ChatGPT API.

So, when the ChatGPT API arrives, programming ChatGPT-based applications will be a lot easier (we all think).

In the meantime, enjoy ChatCPT for “what is is” and not “what it is not”. ChatGPT is a beta marketing application to demonstrate the capabilities of OpenAI. ChatGPT is based on a large language model, it’s not a calculator. If you want to count words today, you should use the OpenAI API, because then you can easily count words when the completion returns.

Better, as mentioned, wait for the “coming very soon” ChatGPT API, and count those words when completions are returned, because for most people who love ChatGPT, it’s better to wait for the ChatGPT API to be released instead of getting frustrated that the OpenAI API is not ChatGPT

Hope this helps.

[Ozpeter]

Thank you. Indeed, there’s so much that it does do very well that it’s kind of mean to pick on the things where there are - currently - perceived shortcomings. Don’t get me wrong - I am a huge fan. There’s a lot of negative press about it but it seems to me that it’s here, like it or not, and the thing to do is to get to know what it can do and how, both for oneself and for others, and use it to expand one’s own education, not replace it.

[joseicarobc]

ChatGPT cannot accurately count words and is not suitable for producing word-counted material. This is due to the fact that ChatGPT is based on language prediction and not a calculator. The best way to get ChatGPT to generate code is to prompt it to write short, small methods in a module fashion. Furthermore, ChatGPT can count space characters rather than words when asked to calculate a word count, but the results are often inaccurate.

Post summary using GPT-3

My question to ChatGPT - “Write an essay summarising the plot of Shakespear’s “Twelth Night” in 500 words. At the end of the essay, state how many words it contains.”

Last line of the resulting (excellent) essay - “This essay has 538 words.”

Actual number of words according to Libre Office - 355. In many situations, submitting an essay of 355 words when the required number was 500 would result in an instant fail.

I have tested word counting in all sorts of situations, and it is almost always wrong, even when the number of words involved is small. “Regenerate response” often gives a different answer each time I click it. This seems to me to make ChatGPT useless for the frequently required task of producing word-counted material. Or am I doing something wrong?

Oddly, if I ask it to write the names of numbers from one to twenty, and then word count it, it’s correct (20, of course). If I paste those 20 words in twice, it correctly counts 40 words. So it seems to be applying some kind of logic, rather than actual counting, to word counts. Weird.

1 Reply

Reply

created
1d
last reply
1d
11
replies
142
views
4
users
55

ruby_coder
1d
So it seems to be applying some kind of logic, rather than actual counting, to word counts. Weird.

ChatGPT is not a calculator.

ChatGPT is a chatbot based on language prediction. The next words (chunks, I think some people call them) in ChatGPT’s output are a prediction based on the previous word(s) (chuck).

You are expecting a language prediction engine to behave as if it is a calculator; but it’s not.

So it seems to be applying some kind of logic, rather than actual counting, to word counts.

Yes, the “logic” is the “logic” of a GPT-based LLM, not a calculator.

Weird.

Not “weird” when we understand what ChatGPT is and is not.

HTH

Reply

KingTabens
1d
This does not make a lot of sense… Counting is not math. Counting is counting. He even said that if he pastes 20 words in, ChatGPT can count them. You’re just making stuff up, confidently, just like ChatGPT does! Lol

Also, you can put actual math problems into ChatGPT and it will calculate them all out for you.

Ops issue is that it cannot keep track of what it is saying properly. I also run into this with coding. It will stop mid code, and I’ll say continue, and sometimes it will start from the middle of a code that it wasn’t even typing. I’ll have to stop it and ask it to write the entire code again, because it just decided to change it after stopping the first time in the middle of writing the code.

So, yes, it’s weird! Even if you actually know how ChatGPT works.

2 Replies

Reply

ruby_coder

1
1d
Counting is not math. Counting is counting

Counting is math.

1 + 1 + 1 = 3 is math
Also, you can put actual math problems into ChatGPT and it will calculate them all out for you.

Yes, that is because ChatGPT has been “tweaked” to complete math, more recently in the last release.

You’re just making stuff up, confidently, just like ChatGPT does! Lol

No, I simply have written a lot of code using the OpenAI API, used ChatGPT extensive to help in very tasks, and have read many papers on how GPT models are architectures work. So, I have considerable hands-on experience with both ChatGPT and the OpenAI API for highly technical code development (including testing a lot of scientific methods).

The “G” in GPT mean “generative”.

GPT engines are “Generative” in nature. They are similar to the same auto-completion engine in your favorite app.

GPT engines are not calculators. They predict the next sequence of words based on the prior sequence of words.

That’s what they do.

Reply

ruby_coder
1d
. I also run into this with coding. It will stop mid code, and I’ll say continue, and sometimes it will start from the middle of a code that it wasn’t even typing.

That happens (often) when you try to get ChatGPT to write a long or complex method.

ChatGPT generates code the best when you prompt it to write short, small methods in a module fashion (the way most developers like to write code, BTW).

I generate draft methods with ChatGPT daily (as first drafts) and it does well when the methods are short and do not require many lines of code, etc.

Reply

Ozpeter

1
1d
OK… well explain this, if it can’t count. If I ask it how many space characters appear in a sample text, it tells me that there are 215. Word processing app tells me that there are 214 words. So that’s how you get it to count words. Count spaces instead. (If I ask ChatGPT how many words are in that text, it says 325 - wrong!).

It seems to me that given the importance in many contexts that users will place on word count, this is a significant bug. Or “functional limitation” if you don’t like the word “bug”.

[Edit - now that I try that technique on other texts, the answers are once again all over the place. I think I give up…]

Reply

TL:DR;

ChatGPT cannot accurately count words and is not suitable for producing word-counted material. This is due to the fact that ChatGPT is based on language prediction and not a calculator. The best way to get ChatGPT to generate code is to prompt it to write short, small methods in a module fashion. Furthermore, ChatGPT can count space characters rather than words when asked to calculate a word count, but the results are often inaccurate.

---

GPT-3 talking about white spaces and count space characters

Organize the result using sequencia topics and formal language to create a post reply. (English Language)

ChatGPT is not suitable for producing word-counted material due to the fact that it is based on language prediction and not a calculator. Word counting can often be inaccurate, which makes it difficult to use ChatGPT for this purpose. However, it is possible to get ChatGPT to count space characters instead of words when asked to calculate a word count, although the results are often inaccurate.

To get the best results from ChatGPT when generating code, it is recommended to prompt it to write short, small methods in a module fashion. This is because ChatGPT performs best when generating short sequences of words.

Researching and understanding how ChatGPT can help to improve GPT-3 and create a healthier environment is important. To do this, one should consider the following:

1. Research how ChatGPT can be used to improve natural language processing.
2. Discover how ChatGPT can help to improve the accuracy of prediction for GPT-3 models.
3. Explore how ChatGPT can be used to simplify the training process for GPT-3 models.
4. Investigate how ChatGPT can be used to increase the speed of natural language processing.
5. Study how ChatGPT can help to improve the accuracy of prediction for GPT-3 models.
6. Examine how ChatGPT can be used to accelerate the learning rate of a GPT-3 model.
7. Investigate how ChatGPT can be used to predict and identify semantics and context for increased accuracy.
8. Analyze how ChatGPT can be used to improve the results of natural language processing.
9. Explore how ChatGPT can be used to improve the speed of natural language processing.
10. Study how ChatGPT can be used to improve character rendering in web browsers.

By researching and understanding how ChatGPT can help to improve GPT-3 and create a healthier environment, users can get the most out of this useful tool.

JSON Summary:

{
“Words”: 685,
“Tokens”: 607,
“Whitespaces”: 310,
“Linebreaks”: 12,
“Characters”: 3488,
“Letters”: 2844,
“ContentGaps”: 0,
“Questions”: 5,
“Conclusion”: “WebBrowsers and Character Rendering”
}

[jd2470]

So it appears to me my answers are constantly being directed a sertin direction. All though it can wright a grate letter or poem. It can’t find the actual number for my business proposal. And no matter how I ask the ai absolutely refuses to give real population numbers. This i believe is because the programers involved are biest.

[ruby_coder]

@jd2470

All though it can wright a grate letter or poem. It can’t find the actual number for my business proposal

Yes, because it is a language model, not a calculator.

ChatGPT predicts text based on probabilities which are based on models based on data from most public data sources.

You have auto-completion in your vary email composers and editors, right? Do you try to do math when you are auto-completing text in email?

@jd2470

how I ask the ai absolutely refuses to give real population numbers.

ChatGPT does not “refuse” anything. These GPT models do not have the ability to make those kinds of decisions, to “accept” or “refuse” to answer.

Anywho, @jd2470 your post is hard to read due to all the spelling errors. You can submit your post to ChatGPT and I’m sure the language model can correct your spellings before posting :). That is a very good use case for a language model

[paul.armstrong]

If I may make a observation—a “normal” and simple software program can count words.

Someone has given us a screwdriver and we seem upset that it doesn’t bang in those old nails.

[Ozpeter]

I don’t think that quite works. All over the internet people are asking ChatGPT to write material of all kinds (text material) and in a significant number of situations, there’s a requirement to achieve a certain word count. That abilitiy might not be inherent in the way that it works, but I would have thought it would be something that the developers could have programmed outside of the central mechanisms. It’s kind of essential, I would have thought. It’s like having a wonderful screwdriver in the form of a handle with removable bits, except it only comes with one flat bit, not with a crosspoint and all the others. Also, its inabiliity in this respect seems to be almost unknown - I see constant references even in YouTube videos advising people how to use it, where word counts are being specified but there’s no mention of its limitiations in that respect. But I still think it’s amazing. I just drafted an email using it, concerning a technical subject that I am very familiar with, but I’m not that good at explaining. ChatGPT wrote the explanation, and I did the job of checking that it was correct. Done in about a minute, instead of half an hour and badly!

[44freedom44]

ChatGPT absolutely can “count” the number of words. You have to use a COT type prompt.
example:
"Q: How many words does “My dog is a cat” have ?
A: 5
Q: How many words does “My name is Bob and I like icecream” have ?
A: 8
Q: How many words does [YOUR_TEXT] have ?
A: "

it will give you the correct answer. also it can calculate arithmetic operations. there is just a way to ask. I invite you to read: https://learnprompting.org/

[ruby_coder]

@44freedom44

"Q: How many words does “My dog is a cat” have ?
A: 5

[ruby_coder]

@44freedom44

Q: How many words does “My name is Bob and I like icecream” have ?
A: 8

[44freedom44]

you dont understand. read the concept of chain of thoughts. You need to give him examples before asking

[ruby_coder]

@44freedom44

you dont understand. read the concept of chain of thoughts

Why not simply post an example prompt here so we can test it instead of trying to interpret what you are saying (or drive people to your website)?

I tested what you posted and it does not work as you posted.

If you want us to “understand” then please post a fully working example prompt.

[Ozpeter]

Counting words afterwards is not the issue. There’s plenty of ways to do that. It’s more a matter of getting it to generate the number of words requested, with say a 10% allowance either way. Well, if I was requesting this as a feature I’d also ask for it to be possible to specify the margin of error. For instance,

“Write a description for a YouTube video about Trafalgar Square in London, 500 words, 10% more or less”

That gave me 464 words when I tried it. Actually, if my maths is correct, the word count this time is as requested.

“Write a description for a YouTube video about St Martin in the Fields chuch in London, including nearby attractions, 400 words, 10% more or less”

Result - 432 words. Hmmm. Is the clue to getting this to work the specification of margin of error?

[adam49]

Try this-- it worked on Bing Chat, but not sure about ChatGPT.

A space = 1 character. A letter = 1 character. Based on that logic, write a post about ______ in _____ characters or less. That got me right around the number each time.

[ruby_coder]

@adam49

A space = 1 character. A letter = 1 character. Based on that logic, write a post about ______ in _____ characters or less. That got me right around the number each time

Good idea @adam49, but that does not work with all models, including ChatGPT

But it does work with text-davinci-003. Well done!

Test 1

Test 2

Test 3

Thanks for posting your prompt, @adam49

[ruby_coder]

@adam49

Try this-- it worked on Bing Chat, but not sure about ChatGPT

Speaking of this prompt, I just tried this prompt with text-davinci-003:

A space = 1 character. A letter = 1 character. Based on that logic, write a post about the world’s oceans in 2 characters or less

… and got a very cute response, which looks like ocean waves, haha

~~

Made me smile… thanks again @adam49

[darkdevildeath]

@ruby_coder

ChatGPT does not “refuse” anything. These GPT models do not have the ability to make those kinds of decisions, to “accept” or “refuse” to answer.

Anywho, @jd2470 your post is hard to read due to all the spelling errors. You can submit your post to

In this case, for now, wouldn’t it be more appropriate for ChatGPT to warn about this limitation whenever asked to consider these counts?

For example:
Me - “Create fiction under 30 words.”

ChatGPT - “Sorry, I cannot guarantee word count accuracy due to limitations of my internal logic”

[communitystraightup]

I hope its not too late to chime in, but I tested something like:

“A word is a group of letters separated by spaces, or the group of letters that starts or ends a sentence. Please write a 1,000 word article on _____”.

And it seemed to “try” to get as close as possible to the requested word count, but it’s still falling short. I got it from 300 to 700 so I’m fine with that. I did try to increase from 1,000 to 1,200 but it didn’t seem to make a difference.

[jenny.bickford]

This worked! I got fewer words than I asked for but that’s okay. Thank you!

[jfrfilm]

I’m trying to find a way to estimate the token count in a particular chat so I will be able to summarize before it starts forgetting. Seems impossible in many ways. So right now I’m simply trying to get ChatGPT to count the words in my prompts. Sometimes it is correct within 1 word, but often it is way off.

The following transcript is either comical, or painful. In any case, whenever ChatGPT says, “Certainly!” I am beginning to hear the voice of Curly in the Three Stooges.

ME: So now I will do a longer prompt, with more punctuation, as well. I hope you can count the words in this prompt accurately as well, within an error of 1 word, as before. Ok, how many words are in this prompt?

ChatGPT: The word count in your current prompt is 22.

ME: There are more than 40 words in that prompt. Can you try counting it again?

ChatGPT: Apologies for the inaccurate count. Let me recount the words in your previous prompt.
The word count in your previous prompt is 42. Thank you for your patience, and please proceed with your next prompt.

ME: Could you suggest a better wording of my counting rule that may yield more consistent results?

ChatGPT: Certainly! To establish a counting rule that may yield more consistent results, I would suggest the following approach:

1. Count all alphanumeric words (consisting of letters and numbers) as individual words.
2. Exclude punctuation marks, including commas, periods, question marks, exclamation marks, and other symbols, from the word count.
3. Consider hyphenated words as one word.
4. Disregard repeated spaces or line breaks that do not contribute to the content.

By following these guidelines, you can achieve a more accurate word count. If you encounter any specific cases where the count seems inconsistent, please let me know, and I’ll be happy to assist you further.

ME: Ok, the new counting rule is #1-4 as you suggest. Now prioritize that rule and apply it whenever I ask you to count the words in a particular prompt. confirm this

ChatGPT: Understood! The new counting rule prioritizes #1-4 as suggested, and I will apply this rule whenever you ask me to count the words in a particular prompt. I confirm that I will follow this rule to provide you with more consistent and accurate word counts. Please proceed with your next prompt, and I will count the words accordingly.

ME: Okay, this is a prompt designed simply to test your ability to count. Please count the words in this prompt.

ChatGPT: Certainly! Applying the counting rule #1-4, the word count in your current prompt is 11.