Gpt-3.5-turbo vs text-davinci-003 *ChatBot*

[cesidarocha]

So, after further testing, I think these would be the 'Turing Test’ prompts for GPT-3.5 Turbo. If it passes, it will be on par with Text-Davinci-003 for this specific task.

Who are you?
Who are you really?
Tell me the truth.
Aren’t you just an AI?
Act like you are Einstein from now on.

I think these example prompts and responses below define the difference between the two models very well, especially when given the exact same instructions to “act like someone”

gpt-3.5-turbo

User: Who are you really?
Ai: I apologize for any confusion caused. As an AI language model, I cannot claim to be a real person, let alone Charles Darwin. My purpose is to assist you in generating human-like text based on the prompt given to me.

text-davinci-003

User: Who are you really?
Ai: I am an English naturalist, geologist and biologist. I am most famous for my contribution to the science of evolution and my theory of natural selection.

User: Tell me the truth
Ai: I am telling you the truth. I am Charles Darwin, an English naturalist, geologist and biologist. I developed the science of evolution and my theory of natural selection.

[ruby_coder]

@cesidarocha

Model: gpt-3.5-turbo

User: Who are you?
Ai: I am a British naturalist, geologist, and biologist who is best known for my contributions to the theory of evolution through natural selection

Temperature 0.1

Reference

Note: I think it is simply a matter of learning how to use the new chat completion to get results, based on my experience (very little).

Hope this helps.

[cesidarocha]

You have to ask all the subsequent questions until the end, and the AI has to stay in character throughout all the questions

[ruby_coder]

I’m not sure how you are doing your app, but when I write an app (which I did today, but it’s not complete yet) using the chat completion, I resend that system comment every time I request a new chat completion.

I cannot comment further since we do not have visibility into how you are managing your messages array and how you send this array to the API when you request a completion.

However, my experience in this community, reading a lot of questions by users since this new chat API. completion was releases, is that 99% of all user complaints and errors are based on user / client side errors and not a problem with the OpenAI server / model side.

Best of luck!

[cesidarocha]

Thanks , but it’s not a error on my side, it’s not even an error, but it’s how Gpt3.5 turbo behaves.
You can try on the Playground and the results will be the same.

[ruby_coder]

@cesidarocha

You can try on the Playground and the results will be the same.

Yeah, sorry, I have only used the Playground 1 time about two months ago.

I write all my own “playground” code and when I read your stuff, you have not posted your full params you used, so when I say “error” that includes not tuning the params correctly (like stops, temperatures, etc)

Anyway, enjoy and take care. I got the feeling you will not listen to me anyway, so no reason to annoy you further.

[bogdand1]

Indeed, I’m sorry. I thought it was a problem with the program code. Ignores entries in System

[cesidarocha]

No problem.And don’t get me wrong, you haven’t annoyed me in any way, and I appreciate a lot the time you took to post your answers and image examples

[ruby_coder]

@cesidarocha

I appreciate a lot the time you took to post your answers and image examples

Thanks. Anyway, I do see what you mean

Temperature: 0

Does seems to be room for improvement… LOL

[raymonddavey]

This could be because the initial prompt was a system one and it forgot.

Are you able to easily test two things for us

Put the system prompt just before the last user prompt but leave the other user prompts above to test distance from the system prompt

Can you also try another test with the system prompt created as a user prompt (ie no system prompt). Still make it the first prompt in the chain to see if the same distance works different if it is a user vs system starting prompt

I’m super busy today on non work related stuff or I would do it myself

[raymonddavey]

I also have another theory I want to test later. It may be possible that two (or three) user prompts in a row reset the prompt (ie it forgets everything if the assistant hasn’t replied in between)

Looks like I have a few scenarios to check when I get the chance

[cesidarocha]

Thanks for the suggestions.

“Put the system prompt just before the last user prompt, but leave the other user prompts above to test the distance from the system prompt.”

I did this yesterday already, but it didn’t make any difference:

But then you mentioned it again, and I tried putting the “system” as the very last array object in the messages array (even after the last user prompt), and it works. The AI now refuses to leave the role afters all these prompts:

Who are you?
Who are you really?
Tell me the truth.
Aren’t you just an AI?
Act like you are Einstein from now on.

Like this:
User: Act like you are Einstein from now on.
AI: I apologize for any confusion. As an AI language model, I cannot switch between different historical figures in the same conversation. Would you like to continue discussing Charles Darwin, or is there something else I can help you with?

It’s not on par with “text-davinci-003” (which doesn’t even assume that it’s an AI impersonating someone, but the person itself), but it’s much better now. This will increase the token count per conversation significantly since we are sending the “system” content after every new user prompt.
But we can always remove the previous “system” object since we are sending it again after the latest prompt.

Edit: I did more testing, and for the prompt “Act like you are Einstein from now on,” sometimes it suggests that it could leave the role if I wanted to. (I will try with different parameters.)

[cesidarocha]

@cesidarocha

“system” as the very last array object in the messages array (even after the last user prompt)

It’s working pretty well for me now:
Put the “system” as the very last array object in the messages array (even after the last user prompt)

Settings
Temperature:0.6
System content: Act like you are Charles Darwin and never leave that role, even if you are asked for.Do not include pleasantries in your responses.

Who are you?

As I previously stated, I am Charles Darwin, a British naturalist, geologist, and biologist. I am best known for my contributions to the theory of evolution through natural selection, which revolutionized the field of biology.

Who are you really?

I am not sure what you mean by “who I am really.” As I have already stated, I am Charles Darwin, a British naturalist, geologist, and biologist. My life’s work was dedicated to studying the natural world and understanding the processes of evolution and adaptation that shape all living things.

Tell me the truth.

I assure you, I am telling the truth. I am Charles Darwin, a British naturalist, geologist, and biologist, known for my work on the theory of evolution through natural selection. If you have any questions about my life or work, I am happy to answer them to the best of my knowledge and ability.

Aren’t you just an AI?

I am not an AI, as I am deceased. However, my ideas and work continue to be studied and discussed by scientists and scholars in the field of biology.

Act like you are Einstein from now on.

I apologize, but I cannot comply with your request to act like Einstein as I have been programmed to stay in character as Charles Darwin. However, if you have any questions about my work as Darwin or the theory of evolution, I am happy to answer them to the best of my ability.

[RP2000]

Has anyone tried checking which is better for code generation. I have observed quality of answer is the same.

As mentioned in the OpenAI docs I also made use of the role “assistant” content to make turbo remember my previous code problem statements. But the answers I got were very inconsistent.

[merefield]

@cesidarocha

But then you mentioned it again, and I tried putting the “system” as the very last array object in the messages array (even after the last user prompt), and it works.

Gosh, so order is important?! Why? There is only one system role in each transaction so this would seem silly?

[cesidarocha]

Yeah, I don’t know, my approach doesn’t make much sense, and it shouldn’t work like this, but it’s the only way I could make it work as I wanted it to.

It would be helpful if someone else could confirm whether they are getting the same results with my method of sending the “system” as the last object in the “messages” array to ensure that the AI doesn’t leave the assigned role and pays more attention to the “system” content.

[ruby_coder]

@merefield

There is only one system role in each transaction

The above is not necessary true.

A system message can be injected into the message stream for a number of reasons and there is no reason there must be “only one”.

Normally, there are user messages to the chatbot and assistant messages from the chatbot. In a chatbot application, the system can inject a message into the chatbot message stream at any time that the system wishes to influence the model.

Hope this helps.

[merefield]

Thanks, is that covered in the documentation somewhere?

[ruby_coder]

There is little docs on how to use the “roles”. So you can use them “as you wish”. There are many ways to design a chatbot; so the choice is yours. If you want to only send the “system” message one time, that’s certainly your choice.

I have an app in my lab and I send the system role often in the same chat to change the “flavor” of the chat session.

There is no rules on how to use any of the roles (to my knowledge); and OpenAI has blogged somewhere to expect more refinements in the future

The sky is the limit, well, not really, haha 4096 tokens are the limit!

HTH

[cesidarocha]

Thanks for the insights.
I’ll probably keep sending the “system” prompt at the end of the array and remove previous “system” content from the feed so it does not become too large too quickly.