Hi all,
My goal is not for my fine-tuned GPT-3.5 model to discover new vulnerabilities, but rather to accurately detect those on which it has been trained. Potentially, we might expect it to identify vulnerabilities slightly different from those in its training data. This would demonstrate its ability to generalize.
In theory, the dataset is divided into three distinct sets: the training set, the validation set (which assesses the model’s performance during training and prevents overfitting by adjusting model parameters), and the test set.
The question then arises:
In my case, is it truly relevant for these three sets to be distinct?
Since the objective is not for the model to detect new vulnerabilities, it seems more appropriate to test it on the data it was trained on.
We would then have the training set and the test set identical.
But what about the validation set?
Similarly, it seems relevant for it to be identical to both the training and test sets. Since the validation set is used at the end of each epoch, the model will have already encountered all the data from the training set when it is pre-evaluated on the validation set.
This is certainly not the usual practice, but could it work here?
Thank you very much in advance !