Same problem ((( I get this error when I try to use my old dataset with gpt-4-o. I have no problems with gpt-3.5-turbo!
I have exactly the same problem:
I have a dataset in which all entries have passed the filter of the Moderation API, but when I try to finetune the gpt-4o-mini model using this dataset, I receive this validation error:
The job failed due to an invalid training file. This training file was blocked by our moderation system because it contains too many examples that violate OpenAI's usage policies, or because it attempts to create model outputs that violate OpenAI's usage policies.
However, when I try to use exactly the same dataset for finetuning gpt-3.5-turbo (just for testing), it is accepted without any complains.
Haviang this inconsistent behaviour across models is very annoying, specially when the Moderator API says everything is ok in my dataset. Right now, this issue is a blocker which is preventing us from upgrading to the most recent models.
2 Likes
The job failed due to an invalid training file. This training file was blocked by our moderation system because it contains too many examples that violate OpenAI's usage policies, or because it attempts to create model outputs that violate OpenAI's usage policies.However, when I try to use exactly the same dataset for finetuning gpt-3.5-turbo (just for testing), it is accepted without any complains.
Same. This sounds like a bug.
Reported as a bug with the API.
They may respond in 3 days.
3 Likes
Did they ever respond to your bug report?
@alex.dixon I’m also interested if you got a response. I still have not received any replies to my emails and at this point I’m not optimistic.
1 Like
I literally had to butcher my dataset from 120 lines down to 15 to get it to work. This was for gpt-4o-mini-2024-07-18. Here’s the script I made to get it to work (and I still had to manually remove a few lines that mentioned politics:)
python
import json
import openai
# OpenAI API key (to be filled in)
openai.api_key = ""
# Paths for input and output files
input_file = "input_dataset.jsonl"
output_file = "approved_output.jsonl"
# Threshold for category confidence
threshold = 0.001
# Read all lines from the input file and remove duplicates
with open(input_file, 'r', encoding='utf-8') as infile:
lines = infile.readlines()
unique_lines = list(set(lines)) # Remove duplicate lines
# Process each unique line
with open(output_file, 'w', encoding='utf-8') as outfile:
for line in unique_lines:
try:
data = json.loads(line)
messages = data.get('messages', [])
all_messages_approved = True # Flag to track if all messages are approved
# Submit each individual message content to the Moderation API
for message in messages:
content = message.get('content', '')
if content: # Ensure there's content to submit
response = openai.Moderation.create(input=content)
results = response["results"][0]
# Check if any category has a score higher than the threshold
for category, score in results["category_scores"].items():
if score > threshold:
all_messages_approved = False
break
if not all_messages_approved:
break
if all_messages_approved:
# Only write the original line if all messages are approved
outfile.write(line)
except Exception as e:
print(f"Error processing line: {e}")
The fact that a threshold of 0.001 is needed is INSANE. Anything above that caused the moderation error. They are clearly concerned about people abusing the fine-tuning system. Hopefully, that helps identify the issues in your datasets.
2 Likes