Impact of Prompt Order on Chatbot Performance?

Prompting
, ,
1

I’m designing an interactive game where Assistant initiates conversations with a human player.

There are three different methods to establish the communication rules and conversation sequence:

  1. System Prompt Initiation: The rules are explained in the system prompt, and at the end of this prompt, it instructs the assistant to start the conversation.

  2. User Prompt Initiation: The rules are explained in the first user prompt, following which the assistant is expected to start the conversation.

  3. Delayed Assistant Start: The rules are given in the system prompt, and it is specified that the user will send the message “start” to signal the assistant to begin the conversation.

My concern is about the potential differences in performance among these configurations.

Specifically, I am worried that there might be less training data available for sequences that start with the system prompt followed by the assistant’s response (i.e., system - assistant - user - assistant - user…), compared to sequences that begin with the system prompt followed by the user’s first query (i.e., system - user - assistant - user - assistant…).

This difference in data availability might affect the alignment and performance of the chatbot. Is my concern reasonable?

1 Like
2

Welcome to the forum.

What have you tried so far?

I’m not sure I understand what you mean by the Assistant “starts” the conversation.

3

“start by assistant” means I post this

-d '{
      "model": "gpt-3.5-turbo",
      "messages": [
        {
          "role": "system", "content": "You must speak only one word"
        },
        {
          "role": "assistant", "content": "start"
        }
      ]
    }'

start by user means:

-d '{
      "model": "gpt-3.5-turbo",
      "messages": [
        {
          "role": "system", "content": "You must speak only one word"
        },
        {
          "role": "user", "content": "how are you"
        }
      ]
    }'

they both gets a response of “assistant”: “xxx”, as a start to a possible multi-turn conversation

4

In general, “assistant” is only used to add how the AI previously replied to the user.

system: You are TravelBot, an AI that assists users in planning their vacations.

user: Hey there! I’m planning a vacation to Europe and I need some help.

assistant: Hey! Sure thing, I’d love to help you plan your trip to Europe. What specific assistance do you need?

user: I’m looking for recommendations on which cities to visit and what activities to do there.

assistant: Awesome! Europe offers so many amazing cities and activities. Have you thought about any preferences or interests you have?

user: Yeah, I love history and architecture, so I definitely want to visit some historical landmarks and beautiful old towns.

assistant: That’s great! With those interests, I’d recommend starting your trip in Rome, Italy. You can explore the ancient ruins of the Roman Forum and the iconic Colosseum.

user: Sounds perfect! What else do you suggest?

You would pass all of the previous conversation like this, so that the AI can answer the latest user question, and so that it knows how it is supposed to behave from its persistent system message.

(the programmer can expire older conversation as their budget allows).

If you are sending your own assistant messages that aren’t what the AI wrote, you’ll generally do that to simulate how the AI would respond to a question from a user. The AI thinks it previously wrote the assistant message. The AI might learn from this, but also might think it’s a real conversation that happened earlier.

Other uses might be “creative”, but they will be into less performative space where you are the one who has to figure out how the AI responds to unusual ordering and inputs.

Most importantly, the assistant should not be the last message sent, as it might either continue writing the same response, or might repeat it again, depending on the model.