How to fine-tune ChatGPT for design comparison?

API
1

I’m working on fine-tuning ChatGPT for a specific use cases and need guidance on how to proceed:

I want to use unpublished works (text and images) as training data.
I’d like to train the model to compare similar CV-designs and determine which one performs better. For example, I have data showing that a CV with an image of the applicant getting a 30% higher invitation than one without an image.
How can I fine-tune the model to provide this kind of comparative analysis when a user makes a similar query?

Specifically, I’m looking for:

  • Steps to prepare and upload custom data (including 2 image comparisons) for
    fine-tuning
  • Techniques to train the model on comparative performance
    data
  • Methods to ensure the model can reference this custom data in
    its responses

I already tried to fine tune it but I only know the way to upload one image (not good for comparison)

{
  "messages": [
    {
        "role": "system", "content": "You are an assistant which rates the design of the users CVs."
    },
    {
        "role": "user", "content": "Hi, I would like to get feedback on my CV."
    },
    {
        "role":"user", "content": [
          {
            "type": "image_url",
            "image_url": {
              "url": "https://www.example.com/link-to-cv-as-image"
            }
          }
      ]
    },
    {
      "role": "assistant", "content": "I rate your CV as 7/10. Here are some suggestions for improvement: ..."
    }
  ]
}

Any advice or resources on implementing these features would be greatly appreciated.

1 Like