Thanks for the detailed answer.
I’m definitely not going to spend thousands of dollars running the experiment on images.
I was just hoping that there was some way to predict from first principles what can be achieved in fine-tuning. I don’t want to empty my bank account on experiments.
Can you share a reference which would help me understand this sentence? “And your fine-tune is only affecting the decoder, not its language understanding.”