Using OpenAI API for parsing emails into structured object

API
1

Hello,

I have an idea, which I am trying to figure out if is possible to accomplish. I would like to create an email parser to some specific structured model. It would be used only when my regular parser fails, I would try to accomplish it with AI.

I have started developing this idea using OpenAI Api and Fine Tuning model. I provided some fine tuning examples of how email format looks usually, some outputs how it expected to be, a successfully parsed emails as examples. It looked fine, but it seems that since my object is pretty nested and complex, AI fails to follow its structure, fails to learn how to parse some complicated fields.

And a question arise, do I have a good vision of how this should be achieved? Is fine-tuning a model is enough to accomplish such task or some extra steps have to be taken?

On top I am doing this on .NET and OpenAI nuggets.

Thank you in advance.

2

You can achieve this by making a good system prompt, I don’t think fine tuning is needed here.

Make very clear on the prompt what the role of the assistant is, and provide real examples of an email before and after being parsed.

You may also want to make an additional call after the email was parsed, with a new prompt, which tells the assistant to ensure that the format is correct.

3

Structure is pretty complicated, sent emails have many different sources and they are from humans so mistakes happen. This model was developed for 10 years at least, highly nested and complicated to understand. I have over 500k of emails, 95% of them I can parse and train my model with. It will be constantly used so I am not sure if regular prompt is good enough. I need something to work and require as less time as possible for future support.

I tried this logic:

  1. Parse emails to correct model and use for fine-tuning:
    I create a batch with 100-200 examples:
new { role = role = "system",content = "You are an AI model trained to extract structured report data from raw emails. Some additional rules: ect.."
},
new { role = "user", content = emailParsedToReadableJson },
new { role = "assistant", content = trainingModelJson }
  1. Afterwards I fine tune with fine tune object:
{
    training_file = trainingFile,
    model = baseModel,
    method = new
    {
        type = "supervised",
        supervised = new
        {
            hyperparameters = new
            {
                batch_size = "auto",
                learning_rate_multiplier = "auto",
                n_epochs = "auto"
            }
        }
    }
};
  1. And then I use this trained model and try to chat with. I ask to export the same email, I parse all the mistakes it has done and I repeat process, try to fine tune again, with providing its mistakes:
new 
{ 
    role = "system", 
    content = 
    "You are an AI model trained to parse emails into structured JSON (ReportData). Your previous response contained errors. Learn from the correction below and do NOT repeat these mistakes."
},
new { role = "user", content = jsonEmail },
new { role = "assistant", content = aiResponseJson },
new { role = "system", content = errorMessage.ToString() },
new { role = "user", content = "This is the correct output" },
new { role = "assistant", content = parsedReportJson }

But after such fine tuning, it keeps making similar mistakes. I am not sure if this approach is not correct or I need to give it more data :smiley: but I would like to see some progress without spending hundreds of dollars.

4

What other problem I have, using such approach, fine tuning model does not seem to learn, it tries to memorize which is not ok for such variety of emails. Maybe someone know, how could I improve learning process?

Usual metrics

  • Training loss: 0.0001
  • Validation loss: 0.0007
  • Full validation loss: 0.0006