Where can I get list of all hyperparameters for tuning GPT 3.5 turbo model

Community
,
1

I want to retrained the previously fine tuned model “gpt-3.5-turbo-1106”.

  1. Where can I get all the hyperparameters in documentation apart from below
  2. Do we have any parameter like “continue_from_previous_version” to carry learnings from previous training

My aim is : After latest training, model should have learning from previous training also.

client.fine_tuning.jobs.create(
training_file=training_file.id,
validation_file=validation_file.id,
model=" Previous fine tuned model ID",
suffix=“gpt_training”,
hyperparameters={“n_epochs”:10, “learning_rate_multiplier”:1.5}
)

2

API Reference - OpenAI API

Is this what you’re looking for?

3

It shows only below parameters for fine tuning.
batch_size,
learning_rate_multiplier,
n_epochs,
suffix,
validation_file

Does GPT API’s have any parameter like “continue_from_previous_version” to carry learnings from previous training ?

4

I think what you’re trying to do might not be what fine-tuning is meant for. It doesn’t add any new knowledge to the model, instead it just changes how it responds/behaves to inputs.

I recommend reading through the full documentation, but here’s the part that explains when to use fine-tuning versus RAG.

1 Like
5

If you want to continue training a new model from one you’ve already fine-tuned, all you need to do is specify the ft:gpt-3.5-turbo:organization:1234567 that is the result of your first fine-tune as your model instead of the base model name.

1 Like
6

yes @_j , I am using first fine-tune model as model parameter instead of the base model name.

7

Then you have solved your need for any other parameter that refers to a prior training.

The hyperparameters are all determined by formula, based on the size of the input. Knowing how to set them can only be intuited by having run your set before and extracting from the job report what parameters were actually used.

The only one that is straightforward is n_epochs - the number of passes through the training file and the number of times you are billed for its tokens.

The defaults seem to be damagingly high for most cases where you still want the model to operate normally otherwise. The training continuation is good for what might be a default of 9 epochs being instead you making multiple models of 2 epoch increments, and determining the qualities of each.

Not specifying the epochs in an API call is like giving a dealer a blank check for “whatever car you think I need”.

1 Like