What's the point of providing a validation file for fine-tuning?

There is an option to provide a validation file to fine-tune the 3.5 model but fine tune events log only include train loss and accuracy with no validation data whatsoever available anyplace.

So it seems that splitting a validation set from training data serves no useful purpose, only reducing the train set. Or are validation data used internally somehow? If so, for what purpose?


This is exactly my question as well. Does OpenAI’s fine-tuning process for GPT-3.5 Turbo use the validation file for training? Does validation loss influence training e.g. for early stopping or dynamically adjusting learning rates?

I haven’t found conclusive evidence on this. However, I ran tests with and without a validation file using a problematic training set of ~50 examples. Both runs performed poorly in real-world tests. The training loss patterns were similar but not identical. Absence of a validation file resulted in a smoother training loss curve.

Given these limited observations, my best guess is that the validation file doesn’t influence training and it’s only for post-training analysis.

I’d love to have a definitive answer.

@_j It sounds like what you’re saying is the validation file is only used for post-training analysis and does not influence the fine-tuned model directly, correct?

Yes, include it or lose it, should make no difference for the learning process (except that the investment in held-out examples can then be realized by having them also be included in training)

1 Like