Fine-tuning chat based on commit deltas

GPT builders
1

Hi, I would like to ask you for advice regarding creating custom chat.

I have my code repository and I create list of deltas for all the commits. I save them as text files where text file name is consecutive number of the commit and content is text description of what was the difference between previous and current commit.

Then I provide the list of deltas explained by OpenAI API (4o) as text to train custom chat. But it’s unable to discuss anything meaningful with me, regarding the project that I’ve described in details, commit by commit, in earlier step.

I train like this:

# Send the files with commit summaries
def load_training_data_from_txt(directory):
    training_data = []
    for filename in sorted(os.listdir(directory)):
        if filename.endswith('.txt'):
            commit_number = filename.split('.')[0]
            with open(os.path.join(directory, filename), 'r') as file:
                explanation = file.read().strip()
                training_data.append({
                    "messages": [
                        {"role": "system", "content": "You are a helpful assistant."},
                        {"role": "user", "content": f"Explain the changes in commit {commit_number}"},
                        {"role": "assistant", "content": explanation}
                    ]
                })
    return training_data

...

# Send training file
training_file_id = upload_file('training_data.jsonl', 'fine-tune')
print(f"Training file {training_file_id}")

# Start fine-tuning
response = client.fine_tuning.jobs.create(
    training_file=training_file_id,
    model='gpt-3.5-turbo'
)

Could you, please, give me some general guidelines on how to approach creating custom chat?

Thank you!

2

You’ll be much better off providing a function “code_repository_search” or “code_repository_query”, than attempting to train by examples on unseen data. The fine-tuning above might make the AI more likely to respond with an explanation instead of saying “I don’t know what you are talking about”, but there is no knowledge backing that behavior up.