I conducted an experiment to figure out how to best address GPTs in their instructions. I gave it 5 different instruction about the meaning of a made up word in a fantasy language and then asked it about it. I addressed it with
“You”, “I”, “The assistant”, “The GPT” and “{name of the GPT}”. I tested each of the 120 permutations 6 times. This is what I found out:
- You is by far the strongest way of addressing a GPT. In over 50% of the cases, the GPT answered with the knowledge associated to “you”.
- The last instruction is by far the most convincing instruction, with the last info being returned 50% of the time.
Here you can find a detailed write-up of the experiment.
12 Likes
Nice test. It would not have occurred to me to use contradicting info.
Very nice! We need for evidence on prompt tuning (at least there has been posted very few research on this in this forum).
This is a fascinating experiment! Interestingly this is also true of humans. We also recall best (and therefore can act on) information at the end and the beginning of a text passage. My take on this is that it may be a feature of the training data where it is simulating how people are writing (i.e. processing) based on text that came before.
Also if you think how you yourself would give or receive instructions best. You and your name would be the most impactful. Granted here there may be some underlying schemas about the name that are not part of the training, but regardless this also aligns with human tendencies.
EDIT: Now after looking deeper into the article, it seems that position 3/5 was still more impactful than the beginning. This does not fit as well to the picture, but I see I may have thought about this wrong. The experiment doesn’t measure just simple recall, where recency and primacy effects should dominate, but also has a component of solving contradiction. In many cases the last “revision” could be the correct one in texts. But this is now pure speculation. Very interesting.
1 Like