What's the point of providing a validation file for fine-tuning?

There is an option to provide a validation file to fine-tune the 3.5 model but fine tune events log only include train loss and accuracy with no validation data whatsoever available anyplace.

So it seems that splitting a validation set from training data serves no useful purpose, only reducing the train set. Or are validation data used internally somehow? If so, for what purpose?

3 Likes

This is exactly my question as well. Does OpenAI’s fine-tuning process for GPT-3.5 Turbo use the validation file for training? Does validation loss influence training e.g. for early stopping or dynamically adjusting learning rates?

I haven’t found conclusive evidence on this. However, I ran tests with and without a validation file using a problematic training set of ~50 examples. Both runs performed poorly in real-world tests. The training loss patterns were similar but not identical. Absence of a validation file resulted in a smoother training loss curve.

Given these limited observations, my best guess is that the validation file doesn’t influence training and it’s only for post-training analysis.

I’d love to have a definitive answer.

@_j It sounds like what you’re saying is the validation file is only used for post-training analysis and does not influence the fine-tuned model directly, correct?

Yes, include it or lose it, should make no difference for the learning process (except that the investment in held-out examples can then be realized by having them also be included in training)

1 Like