Order of fine tuning

API
1

Recently I made a post asking about how the order of information in training effects the responses you get.

You can see that linked here: Location of information in training

In that we determined that the order does effect responses generated by gpt-3. We also determined that it is part of how gpt-3 works and there are ways to structure prompts to make this less apparent but you can’t get rid of it. Me and my partner recently started a project that requires fine tuning so we started trying to figure it out. I once again had the same question: Does the order of information in a fine tuning document effect the responses it will give and if so, how much and in what way. We are new to fine tuning and don’t fully understand it. Thanks for your help!

4 Likes
3

This is correct. It is how you order each example in the file.

1 Like
5

Correct. We also randomly shuffle the examples over and over again, so the order of examples shouldn’t make a difference.

The order of words within a single example will make a difference on the other hand.

5 Likes
6

Thanks, this helps alot. I don’t have to worry about ordering the data so it will understand it which has been a pain with the training, as it would “forget” training in earlier parts.

1 Like
7

Then, Do the API automatically and randomly sample the data from the dataset to construct a batch for each step? i.e., Do I have not to initially shuffle the dataset (that must be shuffled for normal fine tuning) before submitting .jsonl file to fine tuning API?

Thank you for your help!