Fine-tuning GPT-4o on custom python package

API
1

Hi,
I am trying to fine-tune gpt-4o model on custom python package I have written.
My goal is use/fine-tune an LLM model to generate python code based on my package interfaces for a given user prompt.

I have built few data points like below for fine-tuning.

Example:

>> system
You are a helpful assistant

>> user
Create table using this data [[300, 400], [200, 250]]

>> assistant
`python
import mypackage

table = mypackage.create_table()
table.add_columns(['col1', 'col2'])
table.add_row('row1', [300, 400])
table.add_row('row2', [200, 250])

These are my observations:

  1. When fine-tuned model is used as is, without any context, I am getting hallucinated responses, i.e., generated methods are not in my codebase.
  2. When an example (i.e., usage of package interfaces) is provided as context, model is generating more sensible answers.

Is my approach correct for the problem I am trying to solve? Also, Is there any other way around after fine-tuning where I don’t have to pass in all my package interfaces as context, because I have like 80-100 different interfaces and might add more.

Thank you.

1 Like
2

Welcome to the community!

Fine-tuning isn’t the best tool for imbuing the model with any particular information.

RAG might be a better choice for your use-case. Good luck!

1 Like
3

I understand that new information can’t be imbued to the base model with fine-tuning. But building RAG a better approach if my package has many classes and methods? How to provide all this information as context? During RAG approach, I am assuming I should pass few examples as context based on prompt post vector search, What if an important package method is missing in the top-k data points?

My package has many methods to deal with financial data and I want to leverage GPT-4o reasoning capabilities to generate code using this package.
Currently, the base model such as GPT-4o can generate compilable and logical code using pandas library, I need it to do the same using my package.

Please guide me if I am missing something here.

Thank you.

4

Hmm. I’m curious as to why “top-k” is so popular nowadays :thinking:

An “auto-coder” is one of the many holy grails in this space, has been for a while. A lot of blood, sweat, and tears have gone into this, resulting in hype, scandal and letdown. I’m not saying you won’t be able to crack it, but it’s definitely not a trivial endeavour.

I would suggest approaching this (as any LLM based system issue) as a workflow. Identify and characterize your process, codify it, and use the LLM tools to re-create it.