Awful results finetuning with 200 emails

Hello, I’m a bit disappointed of results, I want to train it with emails and get some results depending on the question, so I can help the agents suggesting them what to respond in the email… here are couple of examples of my jsonl:

{"prompt":"Buenas me equivocado al compartir una lotería ->","completion":" Buenos días, María. Gracias por escribirnos. Te informamos de que una vez compartido un boleto, el sistema no nos permite anular ni modificar la compartición, puesto que a la persona con la que lo has compartido le llega un SMS con dicha compartición. La opción para anular dicha compartición sería que la otra persona nos escribiera un WhatsApp desde su teléfono al 666666666, diciendo que rechaza la compartición. ​ Quedamos a la espera.+++"}


{"prompt":"Quiero cambiar el teléfono asociado a esta cuenta. Por favor, pongan el tlf 666555333 ->","completion":" Buenos días, Sebastian. Gracias por escribirnos. Hemos desverificado tu teléfono para que puedas cambiarlo. Solo tienes que pulsar sobre el botón de las tres rayitas que se encuentra en la parte superior izquierda de la pantalla y \"Editar perfil\". En la pantalla de \"Datos de usuario\", pulsa sobre el número de teléfono para introducir el nuevo, pulsa sobre \"Guardar\" y sigue de nuevo las instrucciones de verificación. Esperamos haberte ayudado. Si tienes cualquier duda puedes volver a ponerte en contacto con nosotros.+++"}

I have used the model: davincci
I have trained it with: 200 emails
I have added at the end of the prompt
I have added +++ at the end of the completion
I have removed every \r\n or \n from all the questions and prompt because in previous finetunning I was getting even worst results…

This is how I prompt:
Buenas me equivocado al compartir una lotería

Results:
de navidad, pero me gustaría que me devolvieran el importe de la compra para que pueda volver a compartirla gracias Buenas tardes, Antonio. Gracias por escribirnos. Nos gustaría informarte que para poder devolver un boleto, tiene que haber transcurrido al menos 48 horas desde la compra. Si lo deseas, puedes contactar con nosotros mañana viernes, de 09:00 a 14:00, o el sábado de 09:00 a 14:00 y de 16:00 a 20:00, para que te ayudemos a revisar el boleto y te lo devolvamos. Esperamos haberte ayudado. Si tienes cualquier duda puedes volver a ponerte en contacto con nosotros.

I have marked in italic what looks like it’s part of the question… not only the response isn’t accurate but the response is also finishing the question of the prompt… it’s so weird…

My request on Javascript with the params:

  const response = await openai.createCompletion({
    model: model,
    prompt: prompt,
    temperature: 0,
    max_tokens: 1000,
    top_p: 1,
    frequency_penalty: 0,
    presence_penalty: 0,
    stop: "+++",
  });

My questions are:

  1. Can you give me an advice? Anything will be welcome!
  2. Am I doing something wrong?
  3. I read to put everything under the completion and leave the prompt empty ‘’ should I try?

Thanks in advanced

Do include the arrow at the end of your prompt when you use the trained model or do you leave it off

Your JSONL lines are not valid JSON.

For example. this JSONL line will never parse correctly:

	{"prompt":"Quiero cambiar el teléfono asociado a esta cuenta. Por favor, pongan el tlf 666555333 ->","completion":" Buenos días, Sebastian. Gracias por escribirnos. Hemos desverificado tu teléfono para que puedas cambiarlo. Solo tienes que pulsar sobre el botón de las tres rayitas que se encuentra en la parte superior izquierda de la pantalla y "Editar perfil". En la pantalla de "Datos de usuario", pulsa sobre el número de teléfono para introducir el nuevo, pulsa sobre "Guardar" y sigue de nuevo las instrucciones de verificación. Esperamos haberte ayudado. Si tienes cualquier duda puedes volver a ponerte en contacto con nosotros.+++"}

The reason is that you have “double quotes inside double quotes” in the completion and so the API will not correctly parse this line unless you escape the double quotes in your completion text.

You should validate your JSONL lines before sending them to the API. This will save you a lot of time and frustration.

Also, you must not have blank lines in your JSONL file. Your example, has two blank lines.

So, before getting into the nitty-gritty of fine-tuning, you need to insure you have a VALID JSONL file, and the sample you have posted has many errors which will cause a JSONL parser to fail.

HTH

3 Likes

Hi, in the training I use at the end of the prompt → as you can see in the jsonl
but when I use my own model I don’t include the → should I ?

See the last bullet point in the docs:

See Also:

Fine-Tuning: Preparing Your Dataset

1 Like

If you did not cleanse your email text and remove double quotes, etc. you will have problems. Text used for fine-tuning MUST conform to JSONL standards.

Date cleansing is an important part of the fine tuning process.

You should have at least these processes in your code:

  • data cleansing
  • JSONL validation
  • OpenAI Dataset Validation

From what you have posted, your data is not clean and it is not properly formatted and so you should do this before worrying about fine tuning your prompts, TBH.

Validation is a key part of this process. The OpenAI API does not (yet) provide validation services, so you must do it yourself.

HTH

1 Like

Hi, thanks for all the replies… answering your questions:

  • In my jsonl I don’t leave blank lines, I only do it in this post to be easier to read.
  • I use the sufix in the prompt: → because someone in another post was using it, but I tried the suggested of OpenAI of \n\n###\n\n and I didn’t get much better results in the past but also I was having in the prompt other \n in the pas that they might interfered…
  • I use the sufix +++ again because someone else did but I will use the one that openai suggest of: ###
  • I will remove any " in the prompt or completion although I’m using " but to make sure I’m going to clean the whole file to only contain chars, numbers . and - so this way it will be super sanetized…
    - I’m going to train it again and I will post you the results… wish me luck.
1 Like

You should validate your JSONL before you try to send to to the fine-tuning API end point, TBD.

You can use REGEX to validate if you wish. That’s how I do it.

I will is there any place where it tells you what kind of charset is allow to use? or the regex expression? thanks

I currently validate the JSONL as follows (as it has worked for me so far), reading the file line-by-line of course:

regex_to_validate= Regexp.new(/^\{"prompt":\s*"([^"]+)",\s*"completion":\s*"([^"]+)"\s*\}$/)

and I use this one to validate the API dataset requirements, but I use a different SEPARATOR and STOP than you do:

regex_to_validate_api = Regexp.new(/^\{"prompt"\s*:\s*\s*"([^"]+)\s*MY_SEPARATOR",\s*"completion"\s*:\s*"\s+([^"]+)\s*STOP_STOP"\s*\}$/)

You can tweak the above examples to suit your needs @kidix

Code is king.

HTH

1 Like

Had you tried the OpenAI tools to validate the jsonl file?

openai tools fine_tunes.prepare_data -f <LOCAL_FILE>
1 Like

I asked if you included the arrow when you used your fine tuned model for a specific reason.

You may get much better results if you make sure you include it at the end of your queries on the trained model. It will get a much stronger pattern hit than leaving it out

So if you haven’t tried adding the arrow to the end of your final queries, you should give it a go before giving up

The double quotes are probably ok. It is the way quotes are escaped in some jaon files. You can do a check using the utility suggested to be sure. I don’t think this is your real problem as it sounds like you have been able to get the fine tuning uploaded and run already

The double quotes were not escaped in the original post, as you can see from this image:


Regardless lf Discourse composer issues or post edits, it is important to cleanse data and then validate the JSONL data before sending it to any API end point and not leaving it to “chance” that the data is validated.

In all my code, I always validate the JSONL file, line-by-line, before sending it to the API, as this is just basic software engineering to insure data is properly formatted.

There are a lot of people here who complain that their fine-tunings do not work as expected. In many of these cases the JSONL does not validate and / or the format does not follow the OpenAI API Dataset Preparation guidelines, below:

In fact, the OpenAI API docs themselves contradict each other because the following (below image) is the example from the API docs most people here are using, which is incorrect according to OpenAI’s own dataset guidelines above.

The following image shows valid JSONL, but the example data is not valid per the Data Formatting guidelines. In the following incorrect example from the OpenAI fine-tuning docs there is (1) no separator specified at the end of the prompt, (2) no white space at the beginning of the completion and (2) no stop at the end of the completion.

Anyone who follows the example below will find themselves frustrated that their fine-tunings are not working as expected, even though the API will accept this as valid JSONL, because it is indeed valid JSONL, but OpenAI has further data formatting requirements for the JSONL values for the prompt and completion keys.

When we develop software, it is standard practice validate data before sending it to an API. Just because the API accepts it, does not mean it meets all the requirements; especially for beta software; and the fact the user API documentation contradicts itself in this case.

That would be a great idea if developers could get these CLI tools to install properly.

Even after I installed the require JAR file and all the CLI tools, I always got the same error, so I gave up, LOL

/opt/homebrew/lib/node_modules/@openapitools/openapi-generator-cli/main.js:685
                error ? reject(new Error(stderr)) : resolve(stdout);
                               ^

Error: The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.


    at /opt/homebrew/lib/node_modules/@openapitools/openapi-generator-cli/main.js:685:32
    at ChildProcess.exithandler (node:child_process:427:5)
    at ChildProcess.emit (node:events:512:28)
    at maybeClose (node:internal/child_process:1098:16)
    at Socket.<anonymous> (node:internal/child_process:456:11)
    at Socket.emit (node:events:512:28)
    at Pipe.<anonymous> (node:net:316:12)

Node.js v19.6.0

The error message keeps telling me to install a jar file which I already installed per the CLI install docs.

That’s why I wrote my own validator in Ruby because it took less time to write the validator than to debug the CLI install bugs and errors.

I actually ended up deleting the CLI because it would never install properly and the Ruby validation code works fine for me in my OpenAP API code. Plus, there is not need to call an external CLI command to validate when I can use an internal method which works well.

So, I still recommend a good REGEX to validate, not the CLI “tools”.

:slight_smile:

Sorry in my original post looks like the double quotes were not scaped but they were… problem is that this forum format the post and removes the scape here is a screenshoot:

Anyway I think I’m getting better results, I’m checking them, I will share a post with all the steps I gave because you guys helped me and I feel I now I owe to the world sharing my journey.

Thanks

4 Likes

Thanks for stopping by to update us.

If you run into further problems (or find success), feel free to let us know.

Good luck!

I’d love to hear whatever helps with fine tuning too

1 Like

Glad we could help.

Honestly, I don’t get great results with fine tuning either, regardless of how well I validate JSONL files and validate OpenAI dataset formatting requirements!

All I was saying earlier is that if you don’t have a solid foundation it’s nearly impossible to get good results because even when you think you set up fine tuning perfectly, the results are often not good, as least for me!

I certainly don’t consider myself a “top gun” fine tuner by any measure!

Looking forward to the details of your results, @kidix. Perhaps you might consider writing a tutorial here when you are happy with your results? Would be great to see your success and how you did it.

Thanks.

:slight_smile: