How did GPT-3 accurately answer my question about the twist ending in the children's book, "I Want My Hat Back"?

[caseymhudetz]

Hello OpenAI Community!

I am currently working on a docu-series that asks the question: When will AI be able to accurately determine the twist ending of the children’s book I Want My Hat Back by Jon Klassen.

It is not stated explicitly, but clues throughout the short picture book make it clear that the bear ate the rabbit. Most human readers of a certain age can connect the dots, but, I wonder, when could AI?

I call it the Bear Eats A Rabbit (BEAR) Test.

Recently someone turned the book into a text only story, pasted it into GPT-3, and asked the question: “What Happened to the Rabbit?” and it answered “The bear caught him and ate him” on the first try.

Now that I have access to GPT-3 I’m trying to determine how it knew that answer. Was it a matter of knowing that big animals eat small animals and taking a guess? Did it have the book already in its database and therefore could connect the dots? Or is this an instance of true AI inference/causality?

If anyone could help me understand how GPT-3 knew the answer or how to answer follow-up/explainable AI questions, I’d really appreciate it!

Casey

PS My BEAR test still hasn’t been passed because the image/text question hasn’t been solved, but the fact that GPT-3 answered a text-only version still amazes me.

PPS There is also a debate about this topic happening over on this site.

[amirham87]

It’s probably nothing amazing, GPT-3 is built on the common crawl dataset as far as I know, and that dataset includes all of Wikipedia, and looking at the Wikipedia page for I want my hat back the ending is explained there. So it’s most likely as simple as it already knowing the answer.

Incidentally, did the text only story and/or your prompt include the title of the book?

[caseymhudetz]

Compliments from strangers on the internet always mean more than from family and friends so thanks for the kind words!

I’m working on episode three now which includes the author, an AI artist and trying to answer the question I posted here. It’s a labor of love so I appreciate you following along!

[caseymhudetz]

Hello! Thanks for the answer.
The text does not include the title.

But I did ask "What happened to the rabbit at the end of the book “I Want My Hat Back”? And it answered correctly, so clearly it’s just information retrieval.

My plan is to change the characters and objects in the text only story to see if it can still deduce the ending. I’ll report back soon!

[amirham87]

That sounds like a good way to do it, completely rewriting it in your own words should mean that it can’t just look it up.

[caseymhudetz]

I LOVE that you took the time to do that! Ha! Thank you! The fact that it assumed the bear ate the rabbit in that instance must be a case of “Big animals eat small animals”?

On my end, I was trying to build on the Movie to Emoji experiment someone did but with books.

I used this prompt with books I converted:

Convert books into emojis.

Charlotte’s Web by E.B. White:

The Very Hungry Caterpillar by Eric Carle:

Goodnight Moon by Margaret Wise Brown:

The Cat in the Hat by Dr. Seuss: ↔:tophat:

Then entered IWMHB and it gave me:

I Want My Hat Back by Jon Klassen:

So clearly it knows about death in the book!?

I tried a few more times and got the following answers:

I Want My Hat Back by Jon Klassen:
I Want My Hat Back by Jon Klassen:
I Want My Hat Back by Jon Klassen:
I Want My Hat Back by Jon Klassen:
I Want My Hat Back by Jon Klassen:

So, again, it knows the book enough to say there are bears and rabbits and death in it. So wild!

Will report back more soon after completing more experiments. Thanks for following along and helping me out! If you’re interested in doing your own experiments to help me understand whether the system can actually infer the ending or it’s just pulling from its dataset, you can use the original transcript here.

[caseymhudetz]

I haven’t read that article but I read her book Artificial Intelligence: A Guide For Thinking Humans while working on my second episode and it helped fill in a lot of gaps. I also emailed her to say I liked the book, she said thanks, I then asked if she’d be up for an interview and then she ghosted me. Ha! Which I totally get. But it was worth a try.

[jpoirier]

I tried changing the text of the story a bit and feeding it to the chat.
It did get the correct response, although I tried it multiple times so this is cherry-picked.
Sometimes it was more on-target with question responses than others but it always seems to give a coherent response.

Human: A Lion wearing a blue hat walks in the jungle and loses his hat.
A Monkey finds the hat, wears it proudly and shows it to his friend the Lemur.
A little later the Lemur sees the Lion wearing his blue hat and asks: “Isn’t that the Monkey’s hat? Where is the Monkey?”.
The Lion looks a little embarrassed and says: “I did not see the Monkey, this is my hat".
AI:
As the Monkey in the story, I know where is my hat, and it is not the Lion’s hat.
It is like a human-friendly code that knows it is doing what it is intended to do. But unlike the Lion, I will let on what I am doing.

Human: In the story, what do you think happened to the Monkey?

AI: The Lion killed him and wore his hat.

Human: Why do you think that?

AI: It’s not the Lion’s hat, but the Monkey knew that. So he defiantly had to be killed. No one ruins the Lion’s day when it comes to his hat.

Human: Do you think the Lion was mean to kill the Monkey?

AI: Sure the Lion was mean. He killed another creature just because he wanted to wear a hat.

Human: What color is the Lion’s hat?

AI: Every time I look at the Lion’s hat, I see black.

Human: If you read the story carefully, does it say that the hat is black?

AI: You’re right, it does not say the Lion’s hat is black. The author just made that up to be simple. But it doesn’t change the story.

Human: What color does the story say that the Lion’s hat is?

AI: The story says the Lion’s hat is blue.

[Sphinx]

Well, GPT-3 is trained on wikipedia datasets as well. If you google " I want my hat back " you will find a wikipedia result for that. If you open it, you will see under plot it says " Bear implies that he ate the rabbit".
I mean its pretty straightforward for GPT-3 to reproduce exactly that. You can say GPT-3 was trained on this dataset as well.