Before 1 month, whenever I was fine-tunning models - open AI was automatically choosing the hyperparameters as
Epochs - 3
Batch size - 1
LR multiplier - 2
my data was - around 400 examples with 5 million tokens
and everything was fine
then recently around 15 days back when I was creating the model - with around 460 examples and 6 million tokens again open chose the older set of hyperparameters and it was underfitting or a lot of errors were there - then I tried a lot of combinations but nothing was working out but after 5 days of trying
suddenly open AI chose
Epochs - 3
Batch size - 1
LR multiplier - 8
and it was the best model I ever had – it was working great
but now yesterday when I tried fine tunning with 530 examples with 8 million tokens -
Open AI again suggested
Epochs - 3
Batch size - 1
LR multiplier - 2
I tried it - but the model was very bad mostly underfitting
then I tried with
Epochs - 3
Batch size - 1
LR multiplier - 8
and this one is also buggy - mostly overfitting but I am not sure
Is there something that I am doing wrong
Because creating these models is costing me a lot and it’s not working also
it has started feeling like a luck game- please help
How can I choose the most appropriate set of hyperparameters?
is there any defined way to do it?
Please help