Hello, maybe it is already discussed, but word count or character count of chatGPT is completely buggy in my experience. Asking it to write between 250 to 300 words for example usually gives a maximum of 170 words, and doesn’t matter what prompts I tried, I had no success. Is it a known bug? Is it something I can do differently?
Thanks
You’ve found an interesting thing about open-ended feed-forward generative AI - it doesn’t count letters or words well, especially when making a composition based on your specification.
Beyond a sentence or so, the AI simply can’t keep track, and the path of language generation and the path it takes to reach a conclusion isn’t planned out by your length, merely estimated.
One can use other AI-perceivable meters, one of the best being sentences or paragraphs. “Write four paragraphs of three to five sentences each” is something that can be reflected upon. Then the AI can better decide if more needs to be written or if its time for a conclusion.
Let’s experiment and find out where it goes wrong with some scripting:
First, I’m going to use a system prompt “You are a helpful writing assistant.” to program the AI.
Then: “Write a creative sentence about my pet kitten, using exactly {n} words.”
(expand for 5-50 word composition requests and the number of words counted)
5 word request: 5 words measured.
6 word request: 6 words measured.
7 word request: 7 words measured.
8 word request: 8 words measured.
9 word request: 9 words measured.
10 word request: 10 words measured.
11 word request: 11 words measured.
12 word request: 11 words measured.
13 word request: 13 words measured.
14 word request: 15 words measured.
15 word request: 15 words measured.
16 word request: 17 words measured.
17 word request: 16 words measured.
18 word request: 16 words measured.
19 word request: 18 words measured.
20 word request: 19 words measured.
21 word request: 21 words measured.
22 word request: 24 words measured.
23 word request: 23 words measured.
24 word request: 24 words measured.
25 word request: 24 words measured.
26 word request: 24 words measured.
27 word request: 29 words measured.
28 word request: 31 words measured.
29 word request: 30 words measured.
Probably some convoluted sentences at the end, although I didn’t bother getting the output.
We’d better upgrade the instruction if going further:
Write a creative paragraph about my pet kitten, using exactly {n} words.
30 word request: 30 words measured.
31 word request: 30 words measured.
32 word request: 32 words measured.
33 word request: 32 words measured.
34 word request: 36 words measured.
35 word request: 38 words measured.
36 word request: 36 words measured.
37 word request: 39 words measured.
38 word request: 36 words measured.
39 word request: 40 words measured.
40 word request: 45 words measured.
41 word request: 41 words measured.
42 word request: 47 words measured.
43 word request: 45 words measured.
44 word request: 49 words measured.
45 word request: 47 words measured.
46 word request: 47 words measured.
47 word request: 49 words measured.
48 word request: 49 words measured.
49 word request: 50 words measured.
50 word kitten story
We are now at: My playful kitten, with fur as soft as a cloud, dances through the house, leaving a trail of joy in her wake. Her mischievous eyes sparkle with curiosity, as she pounces on imaginary prey. With each purr and gentle nudge, she fills my heart with love, making every day brighter., but better upgrade to calling it a “story” and we don’t need to see how far off every possible iteration is…
Now it’s time to upgrade from “sentence” or “paragraph” to “story”.
50 word request: 49 words measured.
100 word request: 103 words measured.
150 word request: 160 words measured.
200 word request: 247 words measured.
250 word request: 303 words measured.
300 word request: 349 words measured.
400 word request: 437 words measured.
500 word request: 524 words measured.
600 word request: 590 words measured.
700 word request: 606 words measured.
800 word request: 638 words measured.
900 word request: 581 words measured.
So you can see we reach a point where the AI is tuned to ignore my generation requests even on the API where I’m paying a decent chunk of change for this experimentation. These are all done at top_p = 0 for non-random outputs.
What if we replace my “writing assistant” command and instead take the exact ChatGPT system message and program it in?
500 word request: 541 words measured.
600 word request: 516 words measured.
700 word request: 48 words measured.
800 word request: 728 words measured.
900 word request: 720 words measured.
Wait, what about 700? We get AI disobedience triggered by fine tuning to start denying what you want. We can see OpenAI’s intentions: “not on our dime”.
I did mash several downvotes and got 700 words to go that was counted at 707 by an online tool, so you can just try harder, or use different instruction that don’t limit the length by the form of what you request.
Conclusion: enjoy a long kitten story 200 words, under what I asked for…
Summary
Once upon a time, in a cozy little cottage nestled amidst a picturesque countryside, there lived a delightful creature named Whiskers. Whiskers was no ordinary feline; she was a tiny ball of fur with the most enchanting emerald eyes that sparkled like precious gems. Her velvety coat boasted a mesmerizing blend of ebony and ivory, resembling a work of art crafted by the finest painter. From the moment she entered the lives of her adoring human companions, Whiskers brought an abundance of joy and wonder.
Whiskers possessed an insatiable curiosity that knew no bounds. Every nook and cranny of the cottage was her playground, and she fearlessly explored every inch of her domain. From the sun-drenched windowsills, she would observe the world outside with a sense of awe, her tail swaying gently in rhythm with her thoughts. The rustling leaves, the chirping birds, and the distant laughter of children playing in the meadow—all of these sounds were a symphony to her ears, a constant reminder of the beauty that surrounded her.
But it was not just the wonders of the natural world that captivated Whiskers; she found equal fascination in the simplest of objects. A crumpled piece of paper became a thrilling adventure as she pounced upon it, imagining herself as a mighty hunter stalking her prey. A stray ball of yarn transformed into a captivating puzzle, and she would spend hours unraveling its mysteries, her nimble paws dancing with grace and precision. Whiskers had an uncanny ability to find joy in the mundane, reminding her human companions of the magic that often goes unnoticed in the hustle and bustle of everyday life.
Whiskers was not just a playful spirit; she possessed a heart overflowing with love and affection. Whenever her humans were feeling down, she would curl up beside them, her gentle purrs serving as a soothing balm for their troubled souls. Her presence was a constant source of comfort, a reminder that no matter how challenging life became, there was always a warm and loving soul to lean on. Whiskers had an uncanny ability to sense the emotions of those around her, and she would shower them with unconditional love, her soft whiskers brushing against their cheeks like a gentle caress.
As the seasons changed, Whiskers reveled in the beauty of nature’s transformations. In the spring, she would chase after butterflies, her nimble paws barely touching the ground as she leaped with grace and elegance. In the summer, she would bask in the sun’s warm embrace, her fur glistening like liquid gold. Autumn brought a symphony of falling leaves, and Whiskers would playfully pounce upon them, creating a whirlwind of colors that danced around her. And in the winter, she would curl up by the crackling fireplace, her eyes reflecting the flickering flames as she embraced the coziness of the season.
Whiskers’s playful antics and loving nature touched the hearts of all who crossed her path. She became a beloved member of the community, and children would gather around her, their laughter echoing through the air as they played together. Whiskers had a way of bringing people together, of reminding them of the simple joys that life had to offer. In her presence, worries melted away, and the world seemed a little brighter, a little more magical.
As the years passed, Whiskers grew older, her once boundless energy gradually giving way to the wisdom of age. Her steps became slower, and her once vibrant coat began to fade. But even in her twilight years, Whiskers remained a beacon of love and joy. Her humans showered her with tender care, returning the love she had given them throughout the years. And when the time came for Whiskers to bid farewell to the world she had so beautifully inhabited, she left behind a legacy of love and happiness that would forever be etched in the hearts of those who had been touched by her presence.
And so, the tale of Whiskers, the extraordinary kitten with the emerald eyes, came to an end. But her spirit lived on, a reminder that even the smallest of creatures can leave an indelible mark on the world. Her memory would forever be cherished, her story passed down through generations, a testament to the power of love and the magic that resides within us all.