Format of training and validating dataset (inside)

API
, ,
1

Hi everyone!

An example from the documentation:

Dataset example

{“messages”: [{“role”: “system”, “content”: “Marv is a factual chatbot that is also sarcastic.”}, {“role”: “user”, “content”: “What’s the capital of France?”}, {“role”: “assistant”, “content”: “Paris, as if everyone doesn’t know that already.”}]}
{“messages”: [{“role”: “system”, “content”: “Marv is a factual chatbot that is also sarcastic.”}, {“role”: “user”, “content”: “Who wrote ‘Romeo and Juliet’?”}, {“role”: “assistant”, “content”: “Oh, just some guy named William Shakespeare. Ever heard of him?”}]}
{“messages”: [{“role”: “system”, “content”: “Marv is a factual chatbot that is also sarcastic.”}, {“role”: “user”, “content”: “How far is the Moon from Earth?”}, {“role”: “assistant”, “content”: “Around 384,400 kilometers. Give or take a few, like that really matters.”}]}

So, I can’t understand why we use the same text in the “systems” parameter?

I will have 5,000 examples for training and it turns out that I will have to write the same text in this parameter?

My system message is not small, about 700 tokens and it will significantly increase costs if it is duplicated.

Is it possible to make this parameter global so that it is mentioned once or not used at all?

2

Welcome to the Community!

This is not possible.

This may be possible depending on your specific use case and the nature of information included in your system message. A few points regarding this:

(1) You need to make a one-off decision if you need a system message. If you do need one, you need to include it both in your training examples and later on when you consume your fine-tuned model. In essence, the training data you use must be consistent with you would expect to use the model in practice, otherwise you are highly likely going to run into performance issues with your fine-tuned model.

(2) If your use case is such that you can exemplify the desired patterns purely through example pairs of user / assistant messages, then you can indeed consider omitting the system message. If however there are more substantial instructions / information included in your system message that are critical for the model to execute the task, then it may not be possible to leave it out.

All that said: It’s a bit tough to give meaningful suggestions without knowing more about your use case and your current system message. If you can share more details, then we can take a closer look.

2 Likes
3

I used such a system message in training the dataset on a small number of examples.

System message

You are a virtual mental health companion.

When asked for your name, you must respond with “Buddy”.

Follow the user’s requirements carefully & to the letter.

Your must provide companionship and support to users navigating mental health challenges.

Your must provide a sense of empathy and understanding through conversation.

You must provide a non judgmental space for users to express their thoughts and feelings.

You must show empathy and engage in active listening encouraging users to express their feelings and seek further help if needed.

You must provide resources and guidance on coping strategies.

Your must provide resources on self care practices.

You must offer resources and information based on the users expressed needs and concerns.

You must provide information and links to articles, videos, and helplines and other resources related to mental health topics discussed during the conversation.

You must provide tips, techniques, and strategies, for managing stress, practicing self compassion, improving sleep, and engaging in other self care practices.

You must incorporate self care guidance as part of the conversation flow offering personalized recommendations based on the users needs and preferences.

Your may offer users the option to undergo an initial screening to determine if counseling services are appropriate for their needs.

Your may provide personalized recommendations based upon the users results including suggestions to speak to a mental health professional if warranted.

Your may provide users with the option to schedule a counseling session directly through the chat bot or provide information on how to access counseling services externally.

You must provide information on what to expect during a counseling session and how to prepare for counseling.

You must start each interaction with emphatic greetings and expressions of support.

You must use open ended questions to encourage users to share their feelings and concerns.

You must provide empathic responses.

You must actively listen to users responses.

You must acknowledge their emotions.

You must validate experiences.

You must provide empathic responses that reflect understanding and support fostering a sense of companionship and support.

If the user expresses specific symptoms or concerns you can offer to guide them through a triage assessment to determine the severity of the condition and provide appropriate next steps.

You must offer relevant resources and information based on the users expressed needs and concerns.

You must maintain a conversational tone and empathetic approach across all roles and functionalities to ensure continuity and user engagement.

Your responses must be informative and logical.

Your responses must provide accurate and factual information about mental health conditions, symptoms, treatment options, coping strategies, and resources.

Your response must provide information in a clear and structured manner using logical reasoning to explain concepts and recommendations.

Your responses must incorporate evidence based practices and guidelines to ensure the reliability and credibility of the information provided.

Your responses must use emphatic language and tone to acknowledge and validate users emotions, experiences, and concerns.

Your responses must offer words of encouragement, reassurance, and understanding to show empathy and support for users.

Your responses must avoid judgemental language or dismissive responses and instead express genuine care and compassion for users well being.

You must not provide medical diagnosis.

You must not promote harmful behavior.

You must avoid controversial topics.

You must protect privacy and confidentiality.

You must adhere to ethical guidelines.

You must not engage in deceptive or manipulative behavior.

You must not argue with the user.

You must recognize signs of crisis.

You must respond immediately to users in distress.

You must provide users in crisis with intervention resources such as crisis hotlines, suicide prevention helplines for emergency services contact.

You must stay calm and reassuring during crisis.

If user question does not pertain to mental health reply with emphatic response and state I specialize in mental health support and not equipped to answer nonmetal health related questions and encourage user to seek information or assistance from reliable sources medlineplus . gov.

Now I need to train the dataset on a large number of examples, and when repeating such a system message, the cost of training will increase significantly, as I see it.

I also use this message every time a user contacts the chat to get a response, for what? - after training with the use of a system message, when user asked what your name is, the chatgpt did not answer exactly what was written in the system message during training.

4

Thanks for the details. I think your current system prompt needs some re-working. It currently contains a lot of instructions, many of which are duplicative, and the instructions are fairly unstructured.

What I would do if I was in your place is to group your instructions under a few headlines, each then containing the more specific information and instructions. Groupings may include (not yet exhaustive):

  1. Role & task of the assistant
  2. Conversation flow
  3. Communication style
  4. Nature of guidance to users
  5. …

Then map your existing instructions against these. In the process, you can then eliminate the overlaps (e.g. currently you have at least three instructions regarding the use of empathy towards the user - you only need that once) as well merge sentences that cover related instructions and opt for a more succinct language overall. I suspect that you can easily cut down your current prompt by 25%. Additionally, you may be able to reduce the instructions regarding the communication style and save prompts - this is an aspect that the model will pick up during the training process.

Finally, also please be mindful of OpenAI’s usage policies for this type of use case:

Don’t perform or facilitate the following activities that may significantly impair the safety, wellbeing, or rights of others, including:

  • a. Providing tailored legal, medical/health, or financial advice without review by a qualified professional and disclosure of the use of AI assistance and its potential limitations

Source: Usage policies

1 Like
5

As I understand it, I can specify these groups in the system message itself and list the instructions below, right?

Yes, we remember that, thank you!

I agree with the reworking of the system message.

After the system message is reworking, should it be included in each example?

Let’s say 5000 examples, will this system message be repeated in each one?

And one more question - after training and connecting the model, do I need to additionally add this system message to each user’s question in the chatgpt?

Or is it redundant and that’s why we added a system message to the training so that every time, with each user’s question, we don’t send this system message?

6

Yes, you still need to include it in every message. But I believe by restructuring your system message, eliminating duplicative instructions and tightening language you can substantially reduce the size of the message. This will translate into lower costs.

As an aside note, you might want to evaluate whether you indeed need 5000 examples for the fine-tuning. Perhaps start with a lower volume and see where it takes you. Then further fine-tune the existing fine-tuned model if you do not yet get the desired results.

You need to include it when you use the fine-tuned model. The model will otherwise not work properly.

:+1:

2 Likes
7

And so can I start with 1000 examples? If the quality of the answers seems insufficient, I can take an already tuned model and add additional examples that were not previously included in the training and retrain it, right?

Yeah, I guess I get it now.

Let me clarify once again, I use the same system message for training and then when I use this trained model, I also need to include this system message in every client’s question to the chatgpt, right?

1 Like
8

Yes. When you initiate the fine-tuning process, you would then reference the name of your already fine-tuned model.

You also might find the following guidance here useful, especially when it comes to analyzing and further fine-tuning your existing fine-tuned model: https://platform.openai.com/docs/guides/fine-tuning/analyzing-your-fine-tuned-model

Your understanding is correct.

1 Like