GPT mentions have some interesting unexpected behavior

I have written a Custom GPT for saving notes I take. It requires a user id to distinguish between different people. And it has the ability to create a new user but it asks you first if that’s what you want it to do.

I tried it out by bringing in into a default ChatGPT thread using a mention. When I asked it to save my notes, it skipped my GPT’s “do you want to create a new user” and just auto-called the new user action and then created a new note in a new folder. That is unexpected, because as I said, my Custom GPT Note Taker specifically ASKS you first if you want to create a new user, new folder and note.

But using it as a mention, it happens automatically. Unexpected! And it means I have to update my Custom GPT to handle such situations.

6 Likes

You have to give precise instructions to the bot.

First contact, greet the user, and ask him to store his name.
Second step,
Third step.

This is done in natural language in gpt builider, but you have to be precise in these steps.

Now you also have to tell him what not to do, if for example a bot is a former math expert, and his knowledge base is that, if you don’t tell him what not to do, the user will ask for a food recipe and he will give it.

MY recommendation is that you design your instruction structures in JSON files.

3 Likes

Oh, I already give my bot very precise instructions and also tell it what not to do. And they work perfectly fine when the bot is talked to directly.

But when it is mentioned, it skips some of the instructions, almost as if it is rewriting them. Not sure.

I use Configure and not the Builder because I prefer complete control of the instructions.

Using JSON for the instructions is an interesting idea. Nothing wrong with a little more structure. I’ll experiment with that.

1 Like

It is also useful to think of AI as a response generator. At the heart, that is all it does, it generates a text completion that is reshaped by a series of input placed into its context length.

Why is this applicable? You can change your mindset and instruction orientation.

With your instructions, you are not programming the internal logic of a computer program. You are affecting what the next sequence of tokens to be generated would be.

So you can indeed use logic the AI can follow. But it can be more powerful if you aren’t talking to an entity that doesn’t exist, and aren’t telling it tasks that would operate on some internal mechanism that doesn’t exist.

A response-based programming:

If x - AI will send xxx to action method A.
If y and z after action A - send yyy to action method B,
else assistant composes message text to user

(this is the idea, although the calling of tools should already be trained and enhanced by their own description of when they are useful).

2 Likes

Yes, Based on my recent use of gpts I find that too many similar GPTs, even same name…

89 AutoExpert gpts

Actually, there should be more, I think can add a option,

AutoExpert by XXX (who created as gpt mention)

TLDR: The issue with mentions is the user sees each mention by name, but ChatGPT only sees itself.

People responding don’t seem to understand that you could have very specific or detailed instructions and it just ignores them on an @ mention. I tried asking it about past responses and no matter who I had mentioned, it always think the whole conversation has happened between only you and them. So if it sees itself as not having followed rules so far, it doesn’t see why it should start now.

In your situation, it’s told to do these things when it greets the user, but to itself, it’s already started the conversation and missed the opportunity to greet you, so it just tries to guess what to do next. These are trained on human responses, so like a human that forgot to ask up front, they kind of just run with it and makes a new user rather than interpreting the conversation.

The issue is it doesn’t realize it’s only a mention, it thinks the entire conversation is it’s own. The fix that would probably work for you would be to give it instructions in case it realizes it failed to do something upon greeting the user. You could even try telling it that it will sometimes be dropped into a conversation and tell it to always perform certain steps if it realizes they haven’t been performed yet. You would have to play around with it to see what works best, but this if the approach that works.

If ChatGPT is ever going to work with multiple people in a conversation, this is probably the time to implement it, because giving each GPT it’s own name in the conversation would probably be the easiest fix. They already have work structures, if they made a chat where GPTs only responded when you @ mention them and they saw all the names in chat, they would also respond to the single person chat @ mentions correctly. I have seen examples of this being done with the API, but I don’t know how much work it would be to do it here.

1 Like

Thanks for your analysis and comments. I think you may be right. Hopefully this is all just growing pains and they improve it over time.

Fortunately, it was an easy fix. I added instructions to always ask the user if they have an id before automatically creating a new user. Wasn’t necessary when the GPT was the first in the chat, but was when it’s mentioned.