Hello everyone,
i recently fine tuned a couple of models and used the weights and biases addon to visualize my fine tunes. The problem which i have now, is that there are recognizable steps in th fine tune dataset. This must be some sort of iteration process, but i dont knwo which. Can anyone help me explain this?
Hi Lechnerf,
Do you think there might be a problem or just trying to understand what’s happening? Interpreting the causes of patterns in training loss is a complex topic, but it is quite interesting. I can’t find it now but there was a twitter thread a few weeks ago with detailed analysis of spikes in loss while training a language model.
1 Like
Hi stephenp,
i am generally trying to understand why i have “steps” in both the loss and the accuracy. As my research suggests, this might be due to epochs or batches. But what i wonder is the jump in accuracy between these 4 epochs. The hyperparameters dont get updated between epochs, but only between batches therefore such an increse does not make sense to me.
Thank you i might search through twitter and see if i find something.