I just wanted to share my wild idea, which I came up with while reading BERT and BIGBIRD papers.
Maybe someone has opinions 
Instruct models and maybe future summary models are finetuned to specific tasks.
AFAIK this is currently done purely by finetuning on instruct/summary data
My idea: what if we introduce a new “instruct” or “summary” token - so a task specific token
This could be a way for the general model to be able to perform task specific completions without needing a finetuned version, since the model will jump into “summary mode” on its own when it sees the special token.
Have you thought about this?
There are also a few drawbacks
- increasing vocabulary for task specific tokens (though the number of tasks should not be crazy)
- prompt has to include those tokens, which might be unnatural
- a lot of options to where the summary token should show up in a summary task