Finetuning -- hyperparameter conversions, learning_rate_multiplier values

[logankilpatrick]

@geoff3

Previously, a learning_rate_multiplier value of 0.16 was great, but now that value is way too small, and my training data format is not even close to getting learned – this can also be seen in looking at the training_loss values.

Using the “auto” setting of hyperparameters puts learning_rate_multiplier at 2.0. For my use-case, this is much too large and I can see some clear overfitting when testing. Without doing another full ablation study, how can I think of a good hyperparameter setting for learning_rate_multiplier?

Ideal hyperparmeters will change from model to model, even when your input data might be the same. In general, the default we set are values that our research team has validated work good in many use cases. There are of course outliers but the general approach to find the new ideal parameters is to go and try and train the model iteratively to see how different values effect the loss.