Http 404 issue when fine-tuning

Is there somebody can help me :sob:

I am writing the fine-tuning code, all the parameters are set correctly. I have also checked the jsonl file, which is correct and can be uploaded normally. But when creating a fine-tuning project, an error will be reported.
Here is my code to create a fine-tuning project:

import openai
import logging
# Set API Key
openai.api_key = "sk-XXXXXXXXXXXX"
logging.basicConfig(level=logging.DEBUG)
# Create a fine-tuning task
try:
response = openai.FineTune.create(
training_file="file-XXXXXXXX", 
model="gpt-3.5-turbo", 
suffix="my-model"
)
print("Fine-tune job created successfully:")
print(response)
except openai.error.InvalidRequestError as e:
print(f"InvalidRequestError: {e}")
except openai.error.OpenAIError as e:
print(f"OpenAIError: {e}")

(My api key and file ID are filled in correctly, and the XXX in the code is a demonstration.)

The following is the error message returned after running:

DEBUG:openai:message='Request to OpenAI API' method=post path=https://api.openai.com/v1/fine-tunes
DEBUG:openai:api_version=None data='{"training_file": "file-XXXXX", "model": "gpt-3.5-turbo", "suffix": "my-model"}' message='Post details' 
DEBUG:urllib3.util.retry:Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None) 
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.openai.com:443 
DEBUG:urllib3.connectionpool:https://api.openai.com:443 "POST /v1/fine-tunes HTTP/1.1" 404 None 
DEBUG:openai:message='OpenAI API response' path=https://api.openai.com/v1/fine-tunes processing_ms=None request_id=None response_code=404 
OpenAIError: HTTP code 404 from API 
(<html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
)

My test version information is as follows:

OpenAI SDK version: 0.28.0
Python version: 3.11.5
API Base: correctly set to https://api.openai.com
API Key: correctly set
Model name: gpt-3.5-turbo
File ID: correct and the file status is processed
The result of running print(hasattr(openai, 'FineTune')) is Ture.

At the same time, I also tried the cURL method, the code is as follows:

curl https://api.openai.com/v1/fine-tunes 
-H "Content-Type: application/json" 
-H "Authorization: Bearer sk-XXXXXX" 
-d "{\"training_file\": \"file-XXXX\", \"model\": \"gpt-3.5-turbo\", \"suffix\": \"my-model\"}"

But the following error was returned:

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

I have checked all the steps and can’t find any problem, can anyone help me?

Reading the API reference, and the examples within, can help.

Or we can use code that will survive similar changes to the OpenAI SDK library that have made the (AI generated) “my code” and disabled endpoints you give obsolete.


Have at it, this script will walk through running a file and parameters until you have a completed fine-tuning.

Set your OPENAI_API_KEY as an environment variable. It should not be hard-coded.

Model also upgraded for you.

import os
import requests
import time

# File path to the training data
jsonl_file_path = './finetune-chat.jsonl'

# parameters
SUFFIX = "tune"  # suffix to add to model name
MODEL = "gpt-4o-2024-11-20"
EPOCHS = 3  # training passes through file

# Get the API key from environment variable
api_key = os.getenv('OPENAI_API_KEY')
if not api_key:
    print("Error: OPENAI_API_KEY environment variable is not set.")
    exit(1)

# Define headers for authentication
headers = {
    "Authorization": f"Bearer {api_key}"
}

# Step 1: Upload the training file
def upload_file(file_path):
    upload_url = 'https://api.openai.com/v1/files'
    with open(file_path, 'rb') as f:
        files = {
            'file': (os.path.basename(file_path), f),
            'purpose': (None, 'fine-tune')
        }
        print("Uploading file...")
        response = requests.post(upload_url, headers=headers, files=files)
    if response.status_code != 200:
        print(f"Error uploading file: {response.status_code} {response.text}")
        exit(1)
    file_info = response.json()
    file_id = file_info['id']
    print(f"File uploaded successfully. File ID: {file_id}")
    return file_id

# Step 2: Initiate the fine-tuning job
def start_fine_tuning(file_id):
    fine_tune_url = 'https://api.openai.com/v1/fine_tuning/jobs'
    data = {
        "training_file": file_id,
        "model": MODEL,
        "suffix": SUFFIX,
        "hyperparameters": {
          "n_epochs": EPOCHS,
          "learning_rate_multiplier": 0.8  # scaling from 1.0
        }
    }
    print("Starting fine-tuning job...")
    response = requests.post(fine_tune_url, headers=headers, json=data)
    if response.status_code != 200:
        print(f"Error starting fine-tuning job: {response.status_code} {response.text}")
        exit(1)
    job_info = response.json()
    job_id = job_info['id']
    print(f"Fine-tuning job started. Job ID: {job_id}")
    return job_id

# Step 3: Monitor the fine-tuning job status
def monitor_fine_tuning(job_id):
    job_status_url = f'https://api.openai.com/v1/fine_tuning/jobs/{job_id}'
    print("Polling status every 30 sec, be patient...")
    while True:
        response = requests.get(job_status_url, headers=headers)
        if response.status_code != 200:
            print(f"Error retrieving fine-tuning job status: {response.status_code} {response.text}")
            exit(1)
        job_status = response.json()
        status = job_status.get('status')
        print(f"Fine-tuning job status: {status}")
        if status in ['succeeded', 'failed', 'cancelled']:
            print("Fine-tuning job has completed.")
            if status == 'failed':
                error = job_status.get('error')
                if error:
                    print(f"Error details: {error}")
            break
        else:
            time.sleep(30)

# Main execution flow
if __name__ == '__main__':
    file_id = upload_file(jsonl_file_path)
    job_id = start_fine_tuning(file_id)
    monitor_fine_tuning(job_id)

Thank you very much for your help, I think it is very helpful.

But there are still some problems. I have configured everything else, but after running it, the following error pops up:

D:\train>py train.py
Uploading file...
File uploaded successfully. File ID: file-KdRRcn71CiHLe6D4HYxSyr
Starting fine-tuning job...
Error starting fine-tuning job: 400 {
  "error": {
    "message": "Model gpt-4o-2024-11-20 is not available for fine-tuning or does not exist.",
    "type": "invalid_request_error",
    "param": null,
    "code": "model_not_available"
  }
}

Then I tried to adjust the model, tried gpt-4o, gpt-4, and the same prompt appeared. Then I tried gpt-3.5-turbo, and it seemed to be able to continue running, but the following error still appeared:

D:\train>py train.py
Uploading file...
File uploaded successfully. File ID: file-6Bt1VMh2ffKxcntvJce9qC
Starting fine-tuning job...
Fine-tuning job started. Job ID: ftjob-m2wfKFgHdcyafuTeUdGfu9ba
Polling status every 30 sec, be patient...
Fine-tuning job status: validating_files
Fine-tuning job status: running
Fine-tuning job status: failed
Fine-tuning job has completed.
Error details: {'code': 'invalid_training_file', 'param': 'training_file', 'message': 'The job failed due to an invalid training file. Invalid file format. Input file file-6Bt1VMh2ffKxcntvJce9qC is in the prompt-completion format, but the specified model gpt-3.5-turbo-0125 is a chat model and requires chat-formatted data. See https://platform.openai.com/docs/guides/fine-tuning#preparing-your-dataset for details.'}

Could you help me figure out what the problem is? Thanks a lot!

I understand. gpt-3.5-turbo only supports the format requirements of Chat-formatted data, and prompt cannot be used to write jsonl.

Now I have solved all the problems. Thank you very much for your support.