Hi guys
So I’ve been stuck with a very specific problem for a while now.
I’m creating a platform where I use OpenAI’s API to create structured programs for people. For these programs I have a list of around 1200 things to be used in the program. Each thing has to be written exactly as it’s name on my list, by the OpenAI model, without changes.
The problem I’m having is that the model keeps making changes to these names, either forgetting words, swapping words around, and things like that. And when that happens, I’m unable to match the names on my list with the modified names in the models output, which causes a lot of problems.
Since my promt was around 9000 tokens, I thought the reason was because my promt was too long (I also tried to tell the model in various ways to only use the names on the list in my promt). So I decided to fine-tune. I created a fine tuned model with a traning and validation file on around 30 examples and more that 100.000 tokens. And got these decent outputs after several tries: Training loss 0.0946 Validation loss 0.1487 Full validation loss 0.3442.
In the training files I’m very specific about the model only using the correct names, and in the example programs I’ve also used a big portion of the correct names. Both in examples with and without “weight”.
But now when I use the trained model I’m right back where I started. The output model is of course better at some things which is good, but it keeps using whatever names it wants and not the exact names I gave it in the training files.
What can I do guys?
Simon