One problem I often encounter is that the completion contains large sections of the prompt text.
I have a prompt design that works very well, and I am happy with the completions it provides. However, perhaps one in five times the completion is pretty much an exact copy of work of the ‘shot’ examples in the prompt.
Is there a good way to penalise this and reduce the probability?
Write better prompts, honestly. Your prompt may be too open-ended or unclear, and you may need more framing so that it’s obvious to the model what the next section should be. You can also try turning up frequency and presence penalties, but IMHO prompt engineering should be the first thing to focus on. Also, you should post prompt examples to get better help.
Agreed. Prompt engineering is of utmost importance. Are you using the API?
Thanks for the suggestions. For a bit of background, I am using GPT3 for a fun application, mimicking a satirical character called ‘Train Guy’ who was created by the British Comic Bob Mortimer - see Bob Mortimer's 'Train Guy' is the funniest thing on the internet right now
Here is an example of the prompt:
I’ve transcribed 7 of the Train Guy sketches, and for each extracted three key points that describe the sketch.
I can now generate new sketched based on these three key points as supplied by a user. The results are very good and the output is genuinely funny. However, there are instance (perhaps one in five) where the completion is pretty much a complete copy of one of the multi-shot examples.
@m-a.schenk as you can see, i can’t really use an example from a different domain.
You need just a little bit more setup. For instance, adding another sentence or two to the situation. Say who Jeff is interacting with, how he feels, what he wants, and whatever else is going on. The more fodder you give it, the more creative it can be. Then you can also use a few more examples, since these are very short, plus you can add an END token for stops.
Comment on: complexion
Location: isolated at home
Situation: Jeff needs more brand recognition
Conversation: Hi Col, yeah, yeah, look at your complexion boy. You are grey-abysmal, like a haunted dishcloth. Yeah, yeah, yeah, hilario, I know, I know. I’ve had a few, so it’s alco-lol. Listen, I’ve just had a virtual conflab with Jeff Lynton, the Jeff Lynton. Une problemo significato regarding the pencil cases, yeah, yeah. He’s got fudgey knickers about brand recognition and market placement, yeh. He wants them to be aspirational with a hint of science fiction. So you need to inflate your thought space with ome very hot fart. So quack on. Don’t fear success, fear the urban fox. Have a campaccuccu on me Col. You are as ever a damp crab. Ciao, and bella ontario. See ya Col. {{END}}Comment on: bracelet
Location: isolated at home
Situation: Jeff wants us to on an algorithm
Conversation: Look at you with your little bracelet, your little wrist-bracelet ya. Makes you look like you once had a girlfriend, ya, hilario, I know, I know. That was a bird’s eye potato rofl. I’ve just been talking with Jeff Lynton, the Jeff Lynton. He’s getting anxious trousers about the pencil case designs. Yeah, yeh, yeh, yeah, yeh. He wants us to work on an algorithm, you know that will identify negative variables going forwards. Yeah, yeah, unfortunately my isolation inspiration station has demand crashed, so this one is on you boy. You need to back-comb the living shit out of this hairball in case it unravels, yeah. Good, good lad Col. Listen, have a campaccuccu on me. You are as always a buttery spaniel. Yeah, yeah, ciao, and bella fraciato. See ya Col. {{END}}
I used the Movie Corpus for similar tasks. Cornell Movie-Dialogs Corpus
It takes some massaging but it’s definitely a good corpus.
Thanks - some very good suggestions there, I’ll give them a try.
Prompt design is a fascinating topic. Especially as the output quality isn’t something you can easily measure directly. That coupled with the random nature of GPT3 makes this a fun challenge!