Is there a way to change parameters mid-generation when streaming?
If not, I’d like to request a feature for this.
For example, if you’re generating dialog, with a prompt such as:
Amy: Hey, paul.
Paul: Hey. How are you today?
The AI may generate an answer for Amy, like
Amy: I’m good.
However, that’s no guarantee. It may generate
Paul: Also, I’m not sure if you’re aware of this, but the new album is out.
This can be prevented by appending “Amy:” to the end of the prompt. However, this FORCES the AI to generate a response for Amy, and not come up with narration for example.
Another way, is to use logit_bias to ban the token “Paul”. However, that’ll prevent Amy from speaking his name.
If instead, using streaming the logit_bias could be updated, “Paul” could be banned for the first token generated only, then unbanned for the rest, fixing the problem.
Another (but costly) solution would be to ban the token, then ignore anything but the first streamed token, then do another generation with that token in the prompt. But this would incur a double cost.
Why do you want to abort a completion in this manner? Typically you would use stops for this and submit a second completion.
I don’t want to abort a completion? I want it to not produce unwanted results. Stop sequences won’t fix the problem. And doing another generation would incur cost.
I guess it’s just not clear what you want. You say you want to ban the “Paul” token which sounds like a stop to me. Do you actually just want Amy’s dialog to be longer? I’ve found that GPT-3’s ability to synthesize chat is pretty bad - you can give it very explicit instructions and it will ignore it, so if you want particular kinds of chat you need to use few-shot prompts.
No, I want to prevent the AI from generating Paul’s dialog, but still generate narration or other people’s dialog.
If the prompt is
Paul: Hey.
It may output
Paul: And good morning.
IE.
Paul: Hey.
Paul: And good morning.
I don’t want it to generate dialog for Paul. If I instead make the prompt
Paul: Hey.
Amy:
Then it’s forced to generate Amy’s response. But I don’t want to force that. It should still be allowed to generate dialog for other people as well as narration, eg.
Paul: Hey.
Michael: Hi.
or
Paul: Hey.
Paul looks around the room.
I can prevent it from generating Paul’s dialog by banning the token Paul, but that would also ban the AI from using the token Paul in the rest of the output.
Try banning the token Paul: (with the colon if that works)
Still, your best bet is going to be few-shot since you want very specific outcomes.
Paul: is two separate tokens. And chat is just an example. This is about guidance in general.