Optimizing System Prompts for fine tuning

I’m in the process of fine tuning gpt-3.5

My aim is to develop an automated system for generating briefing papers from agenda items for city council meetings, transforming detailed agenda information into concise, easily understandable briefs.

These briefing papers will be used when a mayor introduces a new item on the agenda at a city council meeting.

I believe fine-tuning is the best strategy for achieving optimal results.I have about 1300 training pairs structured as follows:

{“messages”: [{“role”: “system”, “content”: “Create a briefing paper for a Agenda item”}, {“role”: “user”, “content”: “Agenda item”}, {“role”: “assistant”, “Briefing Paper”:

The input agenda items follow a fixed JSON structure: (Number, Title, agenda summary, recommendation, case presentation, economics)

An agenda item is typically about 1000 words but can range from 500 to 5000 words.

The briefing paper also follows a fixed JSON structure: (Number, Title, Summary, Expected Decision)

A briefing paper is normally about 200 words, but can range from 100 to 700 words.

Question 1: How specific/detailed should my “System Prompt” be?

Bearing in mind not all my real-world training data necessarily meet the criteria of the system prompt

So should I use a system prompt like this:

Generate a briefing paper from the provided agenda item, adhering to the following criteria

  • The briefing paper should be in natural language that is easy to read aloud and in the tone normally used by mayors.
  • The briefing paper has a length of about 150 words but can range from 100 to 1000 words.
  • The briefing paper should not be too technical and contain long words that are difficult to read aloud.
  • The briefing paper is a short and precise summary focusing only on the most central and important points from the agenda. Unnecessary details should be avoided.
  • Minimize the use of numbers and dates unless they are crucial for understanding the agenda item.
  • The content of the briefing paper must not deviate from what is mentioned in the agenda. No additional information or assumptions should be added.
  • The briefing paper may only contain facts directly taken from the agenda. Speculations or assumptions are not allowed.
  • The recommendation from the administration must always be included in the briefing paper. For local planning matters, this point should be particularly emphasized.
  • Reference to Committees: When relevant, specific references to the committees involved in the matters at hand should be included.

Or a shorter, broader prompt like: “Generate a briefing paper from the provided agenda item.”

Question 2: Do I need to use the same prompt as the system prompt used in the fine tuning when I want to generate briefs using the fine-tuned model?

When generating briefs, the system should always use the same prompt for each agenda item, as it is going to create briefing points for up to 50 agenda items.

1 Like

Hi there - welcome to the Community!

A few points regarding your questions.

I think you want to try and find a middle ground between the very long system prompt and the extremely short one. There’s a lot of potential to sharpen wording in the longer one - I think you could likely get it down to one third of the length. Personally, I would place more focus on content related instructions in the system prompt than on style matters. The style should be picked up by the model during the training.

As for the consumption of the fine-tuned model, you’d have to include the same system prompts when using it or at least one that’s reasonably close to the original one.

1 Like

Hey this is really interesting! @Engrob are you able to give updates regarding your work? I’m curious to get your feedback as to the impact of fine-tuning versus not fine tuning a custom gpt.

In regard to the system prompt, I think it would depend on the amount of impact fine tuning would have on the gpt responses. I’m assuming if it’s trained on existing examples of agenda items it should produce responses similar to the format of the agenda items.

You probably better off fine tuning first and then looking into the system prompt. Just my opinion :beers: