Technical discussion: Task specific tokens

I just wanted to share my wild idea, which I came up with while reading BERT and BIGBIRD papers.
Maybe someone has opinions :slight_smile:

Instruct models and maybe future summary models are finetuned to specific tasks.
AFAIK this is currently done purely by finetuning on instruct/summary data

My idea: what if we introduce a new “instruct” or “summary” token - so a task specific token
This could be a way for the general model to be able to perform task specific completions without needing a finetuned version, since the model will jump into “summary mode” on its own when it sees the special token.
Have you thought about this?

There are also a few drawbacks

  • increasing vocabulary for task specific tokens (though the number of tasks should not be crazy)
  • prompt has to include those tokens, which might be unnatural
  • a lot of options to where the summary token should show up in a summary task