In the past, Boris from OpenAI has stated, “Increasing the dataset size will make a much bigger difference, than tinkering with the hyperparameters. My advice would be to leave the epochs at 4, unless you have a very small dataset.”
From my personal experience, larger datasets don’t always correlate towards better results. If you continued A/B testing larger datasets, I believe you’ll start to notice diminishing returns. Personally, I would stick with a few thousand examples and then continue to A/B test different datasets of the same size.