With the embeddings, you should get better results if you correlate to previously known (labeled) embeddings.
So if your input is:
“I do not like football”
And your previous dislike embeddings for the label “Football - Dislikes” are:
“Football is bad”
“Football is dumb”
…
Etc.
Then you correlate your input to all previous labeled embeddings, and select the category corresponding to the highest correlation.
So you can do this with labeled embeddings, instead of a fine-tune, even though the fine-tune isn’t a bad idea either.
The nice thing about the embeddings classifier, is you can add or remove embeddings on the fly, where as there is no easy “undo” operation in a fine-tune.