How to build a conversation classifier with the new Fine Tuning Job API and Gpt3.5?

API
, ,
1

Now that the /classification endpoint is deprecated, my understanding is that the recommended way to build a classifier is using gpt-3.5-turbo and the Fine Tuning Job API. I also understand that the recommendation is to do as much as possible with prompt engineering, before submitting the ground truth examples for fine tuning. Please correct me if I’ve misunderstood!

If that’s is the case, I assume the prompt should contain

  • the data to be classified
  • instructions on how to determine the label and what labels can be applied
  • instructions on the output format

If I’ve understood all that correctly, it would be very useful to see some examples!

Finally, in my use case, I’m actually doing analysis on a conversation between two humans (something like sentiment analysis). For this purpose, should I include the conversation as text within the prompt, or as part of the structured API submission (that starts with “system: “)?

Thanks!

Simon

1 Like
2

I am also wondering the same thing. I’ve been using a fine-tuned Ada model for binary classification and can find little in the way of guides or explanations on how to do this with babbage-002 or gpt-3.5-turbo.

Am looking to functionally replace this with the new models:

fine_tuned_model = openai.FineTuningJob.create(
    training_file=training_file['id'],
    validation_file=validation_file['id'],
    model="ada",
    classification_n_classes=2,
    compute_classification_metrics=True,
    classification_positive_class=' 1 \n'
)
3

No reply from OpenAI - it feels like classification has effectively been removed from the API.

4

There is usually no need to train GPT-3.5 for classification, you simply prompt it with “Given these classes,### {comma_delimited_list_of_classes}###, which of them is most suitable for classification of this text ###{text_to_be_classified}###”

If you were to have text that is not well known by the model, industrial terminology say, they you could build up fine tuning data set with examples of each classification and fine tune on that, or embed your examples into a vector db and retrieve semantically similar context for a given text you wish to be classified.

5

Although not announced and specifically declared, I expect other various models will also be going away with the removal of gpt-3 completion engines and their endpoint January 2024.

A good format for using gpt-3.5-turbo is to program the AI in a system message, and then send the data to be operated on in the user role. Some instruction should also be in the user role, so the AI doesn’t interpret the text as a question it should answer.

messages=[
{
“role”: “system”,
“content”: “”“Classify web forum conversational exchanges between users.
Goal: Identify final replies that are confrontational, rude, or antagonistic.
Site: The Sewing Bee crafts forum.
Output: JSON enum, 1 line. key=‘classification’: values=‘acceptable, confrontational’”“”,
},
{
“role”: “user”,
“content”: “Forum posts: [[[Simon: I like macrame\nPhil:That’s stupid 1970s garbage.]]]”,
}
]

6

That’s excellent advice, thank you!

7

Hey Simon,

And what exactly you’re trying to analyze within the conversation itself?

Subjects?
Human attitude to certain subjects?
Human attitude to opponent?
Human state?
…

The answer to this question will define the approach to take on how to analyze/classify conversation.