Fine tuning model for comparing 2 json

API
, , ,
1

Hi,

I am trying to fine tune a model where I am trying to qualify a record by matching two json objects criteria and requirements.
I am trying gpt-turbo to calculate score or return how many criteria are met with requirements provided

Sample json

criteria
{“name”:“joe”, “industry”:“tech”,“funding”,“1000”}
requirement
{“name”:[any],“industry”:“techonology”:funding":15000}

for above we should have met 2 criteria name and industry
What should be some consideration to keep in mind? As of now I am getting 50% accuracy with 75 training instance

2

Are you not using any prompting? Dont know your ‘real’ JSON but for this one I would probably just use a detailed prompt.

3

I am using quite large prompt with example. I have also tried small prompt but not able to get more than 50% accuracy

4

Can you explain with a few more examples?! I am not sure I totally understand. But I like the challenge. :slight_smile:

5

I’m going to go ahead and say this is not a great use case for a language model, at least not directly.

Why not just extract and compare values programmatically?

1 Like
6

Welcome @shashma

You can use the models to format your data into a specific JSON format with predefined standardized values.

Then, just parse the JSON and evaluate it based on whatever criteria you have locally with code.

7

Okay, here’s the issue with parsing the values and comparing them manually: Some fields in the JSON are not matched one-to-one. For example, if the ‘location’ in the requirements JSON has a value like [“USA, MEXICO”] and the criteria have ‘Boston’ or ‘Guadalajara’ as locations, in that case, they should match.

There are other values like these for ex. pharma should match with healthcare etc.

8

Okay, here’s the issue with parsing the values and comparing them manually: Some fields in the JSON are not matched one-to-one. For example, if the ‘location’ in the requirements JSON has a value like [“USA, MEXICO”] and the criteria have ‘Boston’ or ‘Guadalajara’ as locations, in that case, they should match.

There are other values like these forex. pharma should match with healthcare etc.

9

Few more ex.
criteria
{“name”:“joe”, “industry”:“tech”,“location”,“boston”}
requirement
{“name”:[any],“industry”:“techonology”:location":[“usa”,“mexico”}

In this case answer should be 3 or 100% all fields met the criteria

criteria
{“name”:“joe”, “industry”:“finance”,“location”,“boston”,“role”:“Eng manager”}
requirement
{“name”:[any],“industry”:“technology”:location":[“USA”,“Mexico”,“role”: “manager”}

In this case answer should be 3 or 75% as the industry does not match.

On the flip side gpt-4 is behaving quite well for the calculation of criteria met and comparison but that is just 1:1 comparison. gpt-3.5 is just awful. I have tried short prompt , large prompt but no luck

10

And do you always provide two json to compare? And each pair of json can have different attributes all the time or is it always the same attributes or is it a changing number of attributes?

And so do you expect it to itself decide that pharma is

And just to be sure you need it to answer how many matching arguments?

11
  1. It can have different number of attributes. I would say out of list of 25 attributes we can have any combination.
  2. I always provide 2 json one is criteria and other is requirements it has to meet.
    3.Number of attributes can change.
  3. I want it to decide things like Boston is a city in USA. Pharma and healthcare are similar or under same umbrella.
  4. I want actually score (number of matching args/total args)*100 but I understand it is a LLM not calculator so I am fine with matching args and can calculate score.

Till now I have seen issues where model has hard time keeping track of matching criteria

12

In your Assistant prompt do you have instructions for each of the possible 25 attributes? That would be needed I think for consistent results. And this is certainly a GPT-4 not 3.5 task.

Feel free to email me some examples that are often failing.

13

Hi,

I believe fine tuning is not available for gpt4. I have quite a good result from gpt 4 just from prompt engineering

14

How can i reach you? What is your email address?