GPT-3 can't count syllables - or doesn't "get" haiku

Write two haiku about pizza.

Pizza is great
I love to eat pizza
Pizza is my favorite

1 Like

Try it as a one-shot?

Yeah, it seems to get words that are used in poetry, including rhyming or the typical length of a stanza, but it doesn’t understand syllables.

I have the same issue with a Soneto

It’s true! GPT cannot count. Here’s an example

Task: Count the 1 's.

1000101 = 3
101010000000001 = 4
11011001 = 5
00001 = 1
0010001010001001011111111110001 = 16
1110 = 3
0010101 = 3
000000000001111111111100000000000 = 16 (GPT's answer.  There are actually 11 one's)

GPT failed in a number of other cases too, for example:

11100 = 4
100000000010000100000 = 5
10101= 5

Edit:
Actually it does way better when you separate each digit into a new token

Task: Count the 1 's.

1 0 0 0 1 0 1 = 3
1 0 1 0 1 0 0 0 0 0 0 0 0 0 1 = 4
1 1 0 1 1 0 0 1 = 5
0 0 0 0 1 = 1
0 0 1 0 0 0 1 0 1 0 0 0 1 0 0 1 0 1 1 1 1 1 1 1 1 1 1 0 0 0 1 = 16
1 1 1 0 = 3
0 0 1 0 1 0 1 = 3
1 0 1 0 1 = 3 (GPT's response, correct!)
1 1 1 0 1 = 4 (GPT's response, correct!)
1 1 1 1 0 0 0 1 1 1 1 0 0 1 0 1 1 1 1 1 0 0 0 0 0 1 = 15 (GPT's response, correct!)
1 Like

I asked all of the assistants to “write haiku”. Here are the results:
DaVinci: Beneath the trees | Cool shade and fresh breeze | a perfect summer day
Curie: A quiet evening | Only the moon and stars | And the sound of my breathing
Babbage: A cool breeze in the air | A leaf falls from the tree | A bee buzzing around
Ada: A star-filled night | The moonlight shines | On all those stars | And makes their light dances |To the love that is in me.
None of them are right!

1 Like

Davinci original or text-davinci? Curious how the text-davinci would do.

text-davinci-002. I also used text-curie-001, and so on.

Count the 1 's.

ab11cd = 2 (training response)
abcd111efg = 3 (correct)
hi111jklmn111op = 6 (correct)
111111111111111111111111111 = 31 (wrong, 21 ones)
000000000001111111111100000000000 = 21 (wrong, 11 ones)

25 days later, and not a single admin fixed GPT.