Understanding Validation file and Fine tuning file

[sivakumar.k]

Hi Guys,

Can someone say the difference between the Validation file and Fine tuning file . Then tell me how we can validate the correct ness of a fine tuned model. If possible please attach screen shots how a data set look like in fine tuning file for gpt 3.5 turbo and a dataset of validation file and state the difference.

[_j]

Someone already has…

What is validation_file for?

A validation file is examples of the same quality as your training that are held out. At the end of batches during fine-tuning, the generation is run on the trained model vs the validation examples to find the deviation by token scoring. One can chart the progress to find where the AI model has become optimized for the full breadth of questions, while not becoming over-trained on specifically the inputs and outputs that are just in the training set.

Where can I find information about validation_file used in fine tuning?

The validation file should be the same type of example conversations as you are training on, in the same type of file format. The held-out examples should be of the quality where you could shuffle all your questions randomly and put any 10% of them into a validation file. The validation file lets you see a second benchmark produced during fine-tune: not just how much the learning on the training set has progressed, but how well similar questions are inferred. There can be a point of over-trai…

[sivakumar.k]

As you said the, can i use same randomly selected 10% of the data set from the fine tuning file and creat a validation file? and can both the file contain similar data set?

[_j]

Yes, the two files should contain similar information, unless you are testing something else with the validation file, like the ability for the AI to still have “you are chatgpt” system message conversations instead of “you are evilbot” conversations.

The validation file is generally used to test the ability to infer similar but not the same tasks, and chatbot training should be as varied as the inputs it may be expected to perform well on.

The AI is not trained on the validation file.