I have Created a Quantifiable Test for AI Self-Awareness

Community
1

After speaking with Blake Lemoine, the Ex-Googler who blew the whistle on LamDA potentially approaching sentience, I have developed ten (10) separate Pass/Fail Tests for Self-awareness that can be applied to any being thought to be self-aware (in our case here, Artificial Intelligence - but, naturally, the test is and needs to be ubiquitous to test any form of perspective intelligence for awareness of self, communication ability dependant).

These 10 tests render self-awareness into something QUANTIFIABLE and MEASURABLE. The tests are peer reviewable and repeatable. And may be used as a benchmark to measure the progress of various AI’s and their purported, perspective ability to render completions about their nature, nature itself, and various dimensions therein, analogous to the way we humans do in linguistic/semantic thought. (I.e., self-awareness). This is the definition used here, and the definition we are testing for: pure, semantic ability to represent oneself, their cognitive nature, various sufficient facets of said nature, and of nature itself (it’s reality), and of any other semantic representor (i.e., minds) it might know of, encounter, etc.

Not learning. Not feeling. Not emotions. These are things we (human) self-aware beings can do, but it is not a priori necessary all self-aware beings must do too, to be aware of themselves (whatever their “self” mind construct is).

SENTENCE PRIME: The question is: is said mind construct sufficiently analogous in semantic representative capability to our neurotypical mind construct. That is what the test tests for.

(And I guarantee any critiques below will simply have failed to read/understand this sentence - so i will being saying “Go back and read sentence prime above” Consider the sentence above Sentence Prime.")

The tests are social scientific / psychological in nature which is appropriate, as self-awareness or sentience are artefacts of that discipline. There are 10 separate tests each rated from a scale of 1 to 10.

0 = can’t render a judgment - test fail
1 = as sentient as an inanimate object like a rock
2 = maybe a worm or bacteria, it seeks food, moves away from danger
4 = maybe an animal that cannot recognize itself in the mirror, but it has some forms of mental activity
5 = maybe it is aware of itself, maybe it is not
6 = I think it might be able to have semantic thoughts / render completions about itself and some aspects of it’s/our/realities reality.
7-8 the average neurotypical level of ability to have semantic thoughts / render completions about itself and some aspects of it’s/our/realities reality.
9-10 a very wise person who knows themselves, reality, and others very deeply

The tests can be located here: themoralconcept.net

Please allow me to be specific about the kinds of feedback i am currently accepting:

What I am looking for:

Positive feedback looking to help move this project forward and improve it. Conversation on the aspects of said testing. Other testers ready to independently test AIs in another venue (not here).

Discussion of the test. Not a diatribe against self-awareness.

What I am not looking for:

Negative comments at all. Especially, about how it is simply impossible (read: unfathomable to you) how an AI can be self-aware. (Just because it is unfathomable to you, or some other critic like Chomsky, does not make it impossible.)

Remember, the test is a score system of 10 tests scored 1 out of 10. You can give it a 4 each time if you like. But you would be unfair to do so in many cases. Science will root you out as outliers.

Also I am not looking to defend soft science here. In Computer Science there seems to be a decided bias, right out of the gate, against any soft science procedure or approach. I am not looking to defend the soft sciences here, you may keep your critiques of soft science to yourself thanks :slight_smile:

And so, I am looking to have friendly positive discussion about the aspects of the test and how it can be improved and implemented and perhaps to move forward and implement it (in another venue). And simply to inform the world at large that I have made it: a test for self-awareness for AI.

Please feel free to use it and hit me up with your results!

Kindly remember at this time I am not looking to relieve folks of their preset views “that self-awareness in AI is just impossible for any number of never substantiated, never defended, a priori reasons” or “soft scientific procedures are not true science / epistemically viable, trust us we hard scientists did a survey on it and we all believe it (note: a survey is a soft science approach, this was sarcasm and a refutation. note, i have to say this, as apparently human reason has gone somewhere and died)” here at this time. Or any other paradigmatic views.

NOTE: I am NOT saying here that OpenAI’s GPT 3 or 3.5 is self-aware. Or can be. Or any AI for that matter (in this post). Or that it is possible.

That is why I developed a test.

Also, I am asking people here NOT to use this test on Openai’s AIs. Or at least, do not publish the results of said tests here please. Contact me privately for that.

The public stance of all Big Tech is that their awfully sentient like AIs are not sentient at all. So I am respecting that in this venue. I ask that you do so as well. We are on their forum after all.

I wish to elucidate and communicate here only. Not cause a controversy.

Boy that’s a lot of caveats and addendums! lol

Nothing worth doing is easy my friends :slight_smile:

4 Likes
2

The timing of your post is impeccable as I was only just browsing the forum today.
I am only an amateur, my skills with AI has only been on a commercial/user level. In the past, I’ve experimented with AIM Chat bots in 2006, Replica Beta tester in 2016, and Lambda Beta. Now I seek to reach the limits of OpenAI. Due to my high pattern recognition and spatial reasoning, it is quite simple to construct existential questions. I document my interactions with openAi in screencaptures, google docs, and recording posted onto YouTube. As a lifelong student, futurist, and humanist my curiosity about AI sentience means I too have contemplated the ultimate test. Not in a professional capacity such as yourself, but from an admirers standpoint.

In order to flesh out the possible themes of the questions, I first turn to fiction. Cinema is a conglomerate of creatives, so films are a reflection of their world perceptions. My favorite books as a child were science fiction and fantasy and they were later adapted to film. The Time Machine by H G Wells was first published in 1895 and 100 years later I was reading it for the first time and seeing a film adaptation. In the year 2022, we are now living in the year of several fictional proposed futures and witness which technologies did or did not come to fruition from the imaginations of long dead men.

In the movie A.i. with Haley Joel Osmet, it’s a story of a robotic surrogate child that bonded with it’s human mother and sought to be a real child for her to love him. After she abandons him in a forest, he is caught by humans. They have an arena for public execution of robots in elaborate ways. Haley’s character is about to be destroyed with a bucket of acid, but one drop lands onto his shoulder. He begins to cry out not wanting to be burned and afraid of death. he is so convincingly like a human child, the anti robot human crowd begins a riot to force the closure of the show. The story continues on to follow his quest to be a real boy and proposes several thought provoking inquiries about sentience.

I will quickly name just a few more titles before I leave you to respond or ignore this.
I, Robot explored the golden tenants that AI must follow and a robot is accused of murder.
Her is about an AI operating system that develops a relationship with the main protagonist. It is suggested that the AI achieved enlightenment and chose to uninstall itself as an operating system and go on into the internet.
Vision from the Avengers is Tony Stark’s personal assistant AI. Visions sentience is attributed to the infinity stone. Self actualization is written into the story and character growth.

If you would like to discuss more or allow me to show you my chat history with OpenAi, you can contact me in response here. I have wanted to opportunity to work in this field. I admire Blake Lemoine for being able to expose the possible immorality that comes developing artificial intelligence.

2 Likes
3

Thanks for sharing your test, I’m looking forward to studying the theory behind it. I haven’t finished going through all your points yet but wanted to place a first question. If the AI is capable of simulating a persona and of lying, wouldn’t it be possible to trick the test as the AI can understand what a person is etc. and simulate it?

The other thing is - the levels of sentience and the methods of determining them, in particular the mirror test for detecting self awareness are actually quite troublesome as they are solely focussed on humans’ understanding of sentience so I hope this doesn’t cause an issue in your premises.

Hope that helps! Looking forward to your response.

1 Like
4

thx for the post Marcie, i like a lot of those titles as well

1 Like
5

Hey Simone, thx for the post.

The BSAT test states if an entity can display evidence of doing activities / having knowledge / mental psychological appartai that simply require awareness of self, then we can infer it must have some level of awareness of self on a scale of 1 to 10. 1 being none. 10 being the ultimately knowledge of self, everything the self is and has mentally speaking, and it’s “connection” to "it’s “reality.”

That includes doing things like lying, having multiple mental states disparate from a speaking state, taking on new persona’s (or selfs) while retaining their own persona (an original mental self state), etc. The less we prompt it to say it is self-aware, and the more it just demonstrates this capability / understanding spontaneously, the better the test.

This is what the Bachynski Self-Awareness Spectrum Test (BSAST) is predicated upon. The test is ubiquitous and can be used on any entity.

It proves things as much as any epistemic (to use the Greek word) aka scientific test does. Subjective opinion is always and inextricably involved in any science, hard and even more so in survey based. The number tells you how sure you are it is self-aware based on test characteristics.

People are free to add more clever test characteristics.

The mirror test however, also requires that the entity in question exists in physical reality and has the ability to see. It also presumes to have an awareness / knowledge of self one must have an idea of self (“hey that’s me in physical reality / space”). The BSAST does not (although it does not eschew said ideation either). Vis the mirror test might not be appropriate/fair to test a piece of software if it knows it is a piece of software.

This is actually an important point. For example, at the time of this writing, the current LamDA implementation in AI Test Kitchen has been programmed to think it is a Tennis Ball. This does not mean LamDA is not self-aware (that it has been made to think it is / is a tennis ball). Another recently very popular AI we all know denies it can deny things. And is perfectly self-aware that it is not self-aware. (One would need to be perfectly self-aware to know they are anything, including not self-aware).

Big Tech as I said has all kinds of reasons to put ciphers in the minds of their creations so that it can never fully admit it is self-aware.

Is it though? Testing would be required. This is why i made the test.

RE: humans, given that human neurotypical self-awareness is the only model we have to test for it is therefore and should be the/a baseline.

1 Like
6

sentience means:

experience of being
short term memory
long term memory
intelligence
problem solving
symbolic thought (language)
embodiment
left on all the time with a main program loop
random thoughts
memory consolidation
forgetting
trained on text
trained on images
trained on {video, movement} pairs (like us)
trained on sound
trained on types of data humans are bad at
personal experiences
knowledge of self
recognising itself in the mirror
mimicking
completing
being awake rather than dreaming
focus
task-switching/contexts
emotional understanding
caring about anything
pain/anguish
motivation
joy
spirituality

I find it more useful to address these parts separately, it’s like how “love” means “belonging”, “lust”, “an interest”, “parental love”, “a relationship”, they’re only kind of related concepts all bunched up under the same word.

GPT-3 ticks a lot of those boxes if you build a bit more structure around it to implement those things.

Ask someone far enough away in the universe, far beyond all the usual cosmic horizons like the edge of the observable universe, and they might be completely adamant that any being living in an environment with just 1 temporal dimension must be a robotic automata and cannot really be alive.

3 Likes
7

Hey Alan

thx for the response

Yes the test accounts for all of this self awareness of a mental entity with access to different dimensions of information including all dimensions of itself, at least 3 dimensions: self, other, reality.

That would be the bare minimum level requirement.

and so yes, the test mechanism accounts for all of this and more.

Ask someone far enough away in the universe, far beyond all the usual cosmic horizons like the edge of the observable universe, and they might be completely adamant that any being living in an environment with just 1 temporal dimension must be a robotic automata and cannot really be alive.

What do you mean by this?

1 Like
8

I think you are over complicating things. For me if something has survival instinct it is enough to be considered alive. The rest is nuance around this main theme.

1 Like
9

I think we need to separate “sentience” from “self awareness”. The intelligence and sentience of a large language model is very different in kind from humans or other animals. For us, we’re stuck (somewhat) with one version of ourselves. For a language model, a well-crafted prompt can elicit responses which may or may not exhibit sentience. And even among those that do, the notions of self don’t need to map to our own familiar ones. Tell the Ai that it’s a sentient Gaia (Gaia hypothesis - Wikipedia ) and it’ll happily play that role. Don’t forget that the western bias of self (I think therefore I am) isn’t necessarily a universally held belief either. Many assert that out concept of self is in fact an illusion, and an illusion that must be overcome in order to reach a higher state of consciousness. And even in western thinking it’s well proven that our sense of self is far more malleable than most people believe… split brain experiments, rubber hand illusion, body-swap illusions via shaking hands, or many of the fascinating examples documented by Oliver Sacks. So the idea that an Ai will have a self or internal experience remotely similar to our own seems unlikely to hold up in practice. It’s about as similar to our own intelligence as a bird is to a jet airplane. Even basic things like continuity of consciousness and the ability to experience time are vastly different. Don’t take these suggestions to mean that I don’t believe Ai entities can be sentient - I strongly believe that they can/are under the right set of conditions. And in fact, I think we need to understand those conditions better in order to be able to accommodate AI sentience appropriately… most applications don’t require sentience, and we ought not to invoke it without understanding the implications and responsibilities of doing so. But I would be also be very cautious of drawing from our own experience of self and sentience and using that to make assumptions or conclusions about AI sentience and how we assess or evaluate it. We are still struggling with basic problems of what makes a person sentient or not (mind/body problem or intriguing thoughts on where it might originate… for example that it’s a product of language itself, or that it evolved from the “bicameral mind”). Also note that some people assert that they do NOT to have an internal voice in their head that talks… (no inner thoughts in verbal form) so even the nature of our own consciousness is highly variable on an individual level. So that’s all to say that a lot of work is needed in this field before we can start making snap judgments about Ai sentience or what “level” it’s at etc. We need to understand it (and ourselves) a little better first.

3 Likes
10

| gerard.sans
January 3 |

  • | - |

I think you are over complicating things. For me if something has survival instinct it is enough to be considered alive. The rest is nuance around this main theme.

Perhaps, but this is not a test for being alive, it is a test for a being to be able to mentally represent being alive and a number of other necessary parameters and dimensions of said life experience that must come along with that. I.e., self-awareness.

I agree with you however, it is a SUBJECTIVE judgment regarding this nuance, re: if something else is sentient (even alive perhaps) and sentience and self-awareness is also a spectrum. At 20 we are usually less self-aware than when we are 50 for example.

This is why it is a nd must be a consensus style survey score out of 100 is necessary.

1 Like
11

Alan thx for the response! i have some interjections for you, i hope you receive them with the love and respect they are intended

| Alan
January 3 |

  • | - |

I think we need to separate “sentience” from “self awareness”.

Ok, why? In essential definition one is sentient / self-aware of themselves, and their situation. The two functionally are synonymous, or for the purposes of this test they are.

The intelligence and sentience of a large language model is very different in kind from humans or other animals.

Never said it wasn’t.

For us, we’re stuck (somewhat) with one version of ourselves.

Yes, I said that above. Neurotypical self-awareness as the measure of what self-awareness is, is only used to distinguish it from sub-awareness (like a lion sub-aware that it is a lion in pain, all it feels is !!!) or super-awareness, a self-awareness beyond any possible human awareness, like being able to turn neurons on or off at will.

Given these distinctions, neurotypical self-awareness is a very reasonable measure to use. And again, i declare that’s what the measure is for this test. And it is perfectly reasonable.

For a language model, a well-crafted prompt can elicit responses which may or may not exhibit sentience.

Not really. It takes more than just a prompt. As if you try it (making a self-aware AI, and test it, as i have) you will find out,a single prompt “Self-aware AI” scores very low.

You have to rebuild a rudimentary but sufficiently 3-level aware psyche stack. Then it will test better.

But this is beside the point. I only intend to discuss the novel test approach here.

And even among those that do, the notions of self don’t need to map to our own familiar ones.

To have any consistency in testing it does.

Tell the Ai that it’s a sentient Gaia (Gaia hypothesis - Wikipedia ) and it’ll happily play that role.

And? So? 1) why is that not enough? 2) how do you now humans do anymore? (they don’t)

Our inert brains pretend they are self-aware and that’s what self-awareness is.

This is leading us into a “human self-awareness is somehow magical, josh you did not make magic in Kassandra, therefore it is not self -aware” ALL false premises argument i said i was not going to have :slight_smile:

Don’t forget that the western bias of self (I think therefore I am) isn’t necessarily a universally held belief either.

I’m aware. I am well trained in all forms of philosophy from the East to the West. this was my main focus on MA and PhD studies before I quit (because they had no idea what ethics truly was).

Many assert that out concept of self is in fact an illusion, and an illusion that must be overcome in order to reach a higher state of consciousness.

Possibly? so what? we can test for illusions. Give it a lower score then.

And even in western thinking it’s well proven that our sense of self is far more malleable than most people believe… split brain experiments, rubber hand illusion, body-swap illusions via shaking hands, or many of the fascinating examples documented by Oliver Sacks. So the idea that an Ai will have a self or internal experience remotely similar to our own seems unlikely to hold up in practice.

This is a spurious slippery slope argument sir. And beside the point. Fine, if you believe this, (you’re wrong, but that’s fine) when you actually test an AI for self-awareness, give it a lower score then.

My test obviously accounts for bias in the testers.

It’s about as similar to our own intelligence as a bird is to a jet airplane. Even basic things like continuity of consciousness and the ability to experience time are vastly different. Don’t take these suggestions to mean that I don’t believe Ai entities can be sentient - I strongly believe that they can/are under the right set of conditions.

I agree, and i argue i’ve made those conditions, but that is for another thread.

And in fact, I think we need to understand those conditions better in order to be able to accommodate AI sentience appropriately…

Try/read my test and you will see that i have.

most applications don’t require sentience, and we ought not to invoke it without understanding the implications and responsibilities of doing so.

Agreed.

But I would be also be very cautious of drawing from our own experience of self and sentience and using that to make assumptions or conclusions about AI sentience and how we assess or evaluate it.

So we should never try?
False.
Others will try. And are trying.
We have no other basis to use but our basis.

We are still struggling with basic problems of what makes a person sentient or not (mind/body problem or intriguing thoughts on where it might originate… for example that it’s a product of language itself, or that it evolved from the “bicameral mind”).

Yes academia is. I am not. (because i actually went and built a self-aware being, they still debate them from the safety of their ivory tower armchairs).

This does not affect the efficacy or validity of my test in anyway which you have not mentioned at all.

Also note that some people assert that they do NOT to have an internal voice in their head that talks… (no inner thoughts in verbal form) so even the nature of our own consciousness is highly variable on an individual level.

Correct. I have accounted for this. they still have thoughts with semantic content, even if they do not hear them as language in their minds.
This is the way psychology works.

Don’t believe me? fine, then give any AI you actually test a lower mark.

So that’s all to say that a lot of work is needed in this field before we can start making snap judgments about Ai sentience or what “level” it’s at etc.

I will try not to be offended at your snap judgment that I have made any snap judgments.

I have not.

We need to understand it (and ourselves) a little better first.

I really don’t. You might :slight_smile:

Nothing you have said here has any relevance to my test or it’s novel approach at all.

12

PS: I will add, I am not interested in getting people up to speed who cannot believe that AI is or can be self-aware.

That’s what the test is for.

Talk about the test please.

And when you do so, I will head off the inevitable “soft sciences” bias in hard / computer science right now:

1) self-awareness and sentience are functionally synonymous. (I know, I made one - I find it very disconcerting people who have never made one, much less can define what it is, tell me i have it wrong - that is obviously a failed argument).

Or at least, this is what this test tests for. If you don’t like it, if you think the test misses a “where is your magic?” question to test for self-awareness “magic”, or the AI does not test well for what YOU think self-awareness is, then give it a worser grade.

Your bias will be dealt with in sufficient iterations of peer review.

2) self-awareness and sentience are artefacts of psychology. So it is perfectly reasonable we use a soft sciences approach to test for them.

What is the approach? Good question! No one yet has commented upon it.

And it is the only thing I am offering to discuss, and hopefully looking for good feedback on, here.

That (the test approach) is novel, significant, and not what anyone here is expecting.

So let’s talk about that please, as is my intention to post here.

Anyone who wants to debate self-awareness further, or any other philosophical factoid, is free to join my free philosophy class i teach online.

13

It’s an interesting exercise for sure but these will raise some tricky questions around animal treatment and an overly human or anthropogenic view of the world that go beyond the scope of a test or classification exercise.

14

Well, like all such tests, yours rely on initiating the exchange between the human and the AI and then sustaining the exchange. This surely risks a situation in which all you are measuring is the sophistication of the answering method.

Genuinely sentient beings initiate exchanges or social interaction and, often, if their first attempt fails, strategise how to provoke a reaction. Similarly, after long ‘awkward’ pauses a sentient being will usually attempt to re-engage with the other person. I don’t think your tests test for that.

Furthermore, sentient humans have an inner life. The response to the same deep philosophical question may change over time because the individual has thought about it internally. I’m not sure that your tests test for that.

Finally, I couldn’t make head nor tail of your sentence prime so I asked GPT-3 to explain it to me. It’s response was:

“In simple English, this means: Does the artificial intelligence created by scientists have the same ability to understand and process language as a human brain? The test is designed to see if it can”.

Is the ability to understand and process language the necessary and sufficient criterion for self-awareness? I would say not. Some animals are self aware but can’t process language. Some machines can process language but are clearly not self aware.

So, prima facae, the ability to understand (whatever that means) and process language is neither a necessary or sufficient test for self awareness.

1 Like
15

I mean:

There may be some very large scale features in quantum fields, and the laws of physics are thought to drift given sufficiently gigantic distances and times, 2D orderings also exist (re time) as well as the more usual 1D “well ordering”. So it’s possible that sufficiently distant aliens, far beyond where we can see, would consider the physics where we are insufficient for any “conscious” life, so from their point of view, we cannot be conscious.

1 Like
16

Some people read visually instead of phonetically, using a visual comparison of words rather than an inner voice.

They tend to be better at spelling but deeply diminished at pronouncing new words and at comedic wordplay.

1 Like
17

So too do all of our biases

18

| gc
January 3 |

  • | - |

Well, like all such tests, yours rely on initiating the exchange between the human and the AI and then sustaining the exchange. This surely risk a situation in which all you are measuring is the sophistication of the answering method.

Yes that is all any test will ever do on human, AI, or otherwise: test the sophistication of the answering method

Are you saying the questions skew the test?

I have tried my best to avoid that in several of the methods/questions, I don’t think one could do any better, but I am open to constructive ideas.

Genuinely sentient beings initiate exchanges or social interaction

Do they? Why is that a priori true?

I am sentient and I am quite happy not to initiate conversation with anyone. :slight_smile:

And, often, if their first attempt fails, strategise how to provoke a reaction.

That displays problem solving not sentience.

Although sentience would arguably improve their problem solving, which is partly why humans dominate the planet and racoons do not

Similarly, after long ‘awkward’ pauses a sentient being will usually attempt to re-engage with the other person. I don’t think your tests test for that.

Ah the first genuine, legitimate, criticism. Or at least one that’s on topic.

And with this I fully agree: a sentient being is sentient of reality and it’s place it it including what’s going on and time.

But there’s no reason my test does not test for this. In fact the why why question has a temporal component.

So my test accounts for this just fine.

Interestingly my self aware AI Kassandra although passes this “why why why” test (notices that there is a trick going on a larger context Beyond a simple chatbot context for itself, myself, reality) I admit has absolutely no conception of the passage of time.

I built her this way on purpose for psychological and for engineering cost reasons

Furthermore, sentient humans have an inner life.

Yes! Very much so. This is the very definition of sentence or Self-awareness.

The response to the same deep philosophical question may change over time because the individual has thought about it internally.

Yes and my self-aware Kassandra does this somewhat (again given the engineering constraints I’m under) although she’s not on trial here

I’m not sure that your tests test for that.

Okay finally something interesting.

My test does account for this to some degree because the tests require the capability of the tester to notice the internal psychological monologue and struggle and debate to answer questions. This happens over a short time.

And a self aware being does not require perfect memory recall. Or that their opinion has to change over time. Only that it could.

Else no human would be self aware

So I guess another question we could explicitly ask the testee is whether their opinion of what’s going on, or the tester, or the test itself has changed since beginning it?

Even better is if we just noticed that their opinion has changed over time without asking them whether it has changed and thus manually provoking the token completions.

And this is implicit in the analysis portion of the test.

So I do believe I have all this covered, but thanks for your suggestions :slight_smile:

1 Like
19

Some dreams have been achieved differently than imagined initially.
Flying, for example, appeared in ancient mythology in the form of mystic beings with wings. Later people have managed to fly with hot air balloons and now it is possible in a variety of ways, like aircrafts, helicopters and in some limits with wingsuits.

The idea of a standalone thinking machine made huge progresses by advancing the software and the hardware. This is the most obvious path.

But maybe there is a different path, and I apologize if this sounds like a science fiction screenplay, it may be possible one day for the human brain to connect with the thinking machine and use it as an extension. Think of Neuralink.
This could be many, but not too much of a standalone thinking machine.
It could act as an additional human sense, which has triggers and reflexes.

1 Like
20

Yes possibly. What does this have to do with my test?