Providing context to the Chat API before a conversation

Prompting
, , , ,
1

I’m using the API playground right now to test providing user demographic information to GPT. After reading the Chat Guide I figured the best way to do this with 3.5 is to use a User message. So wrote up a few sentances for the initial user message:

“Use second grade english. I am learning second grade math. Never provide the answer but help me walk through the problem. Use the english language, but keep in mind that my native tongue is spanish. When I give an answer double check that it is correct given the original question, even if my explanation is correct.”

The results are ok but I’m trying to figure out the best way to give GPT this kind of info in order to reduce it’s mistakes. Using multiple user messages seems to make the assitent more likely to ignore the provided context and less empathetic.

Does 1st person or 3rd person language make a difference? Temperature also seemed to make the assitent more likely to ignore the initial context, has anyone else found this to be true?

4 Likes
2

Welcome to our dev community.

System message also plays a big role - though maybe not as much as at the moment. OpenAI is working on improving ChatML (the system/user/assistant method for prompts) to make it even better, I’m sure.

Until then, I would play with putting your “instructions” into the system and then trying the system prompt BEFORE the user / assistant or after… people have had success with both. I think putting system last might weigh it a bit more. (OpenAI has said the weighting on system will likely change, though…)

Another thing you can do is put the instructions into the system message and feed it one user / assistant exchange as an example… so…

System message: (Rewrite your instructions here…)
User message: Make up a user question here.
Assistant message: Make up an assistant answer here that fits your instructions.
User Message: (Append user new input and send to API…)

or…

User message: Make up a user question here.
Assistant message: Make up an assistant answer here that fits your instructions.
User Message: (Append user new input and send to API…)
System message: (Rewrite your instructions here…)

… at least for the first exchange. If you’re going to be chaining them, after you have a few user/assistant exchanges, you won’t need to append the made up one to the beginning.

It’s really all about experimenting for your particular use-case and seeing what works best. Keep in mind, though, that ChatML is likely going to change in the months/years ahead as it improves.

Also, “negative prompts” - ie telling LLM not to do something, is sometimes difficult depending on the size of the prompt, etc. It can easily get confused as not or do not is only a token and it might be surrounded by a lot of tokens that suggest it should do that very thing. So, it’s best to phrase things as a positive sometimes - tell it what you want not so much what to avoid.

5 Likes
3

Hi Paul,

Thanks for the comprehensive response. Your examples for the message sequence is exactly what I’ve been toying with. It’s fun but difficult for me to predict at the moment, I guess that will come with time and more enhancements from the chat api team.

The negative prompt issue seems to be the biggest challenge right now and your explanation makes sense. As you might have derived, this is for a tutor bot. What I noticed is that I can “sandbag” GPT with responses like “I dont understand” until it eventually gives the correct answer. Rarely it stubbornly asks me to work through the problem, which is actually what I’m looking for.

2 Likes
4

Adding a bit more to Paul’s answer, and using your example.
Use the System role for context maintenance. Itemize, numbered is better, concise instructions, mind punctuation as separators:

System:
Please follow the six instructions below:
1. Perform assistance for a US second-grade math learning;
2. Provide a step-by-step process for a solution to any math problem presented in the User role;
3. Do NOT provide any final or partial answer to a math problem;
4. Use US second-grade English language level;
5. Provide Spanish translation in parenthesis of all mathematical terms presented on US second-grade or higher, in problem or solution;
6. Check ALL final or partial answers to the problem no matter the solution process used in the User role.

Some GPT-3.5 versions have a weakened System role importance. In order to make a model-free conversation - it is suggested to do this in the User role:

User:
Follow the instructions in the System role always.
Keep those instructions in context all the time.
...
"second-grade math problem text..."

“I” and “you” work nicely in the User prompt - it is usually expected by the models in a more natural language. In order to keep the conciseness of the instructions in the System role, the pronouns are unnecessary, more robotic command sentences shall be enough.

The models give low importance to the Assistant role concerning to context maintenance.

The User role shall have more detailed information in order to solve the math problem. If you notice that the model “forgets” the context-maintenance instructions, you can repeat the sentence Follow the instructions in the System role... in the User role header to reinforce the System authority over the context of the conversation.

I hope this helps. Please, let us know the results - there are more threads in this forum with these tactics with relative success.

5 Likes
5

Fantastic, excellent guidance as well. I’m interested in how number sequencing and semicolons affect the model’s understanding. It make’s sense to me, as the system prompt is more organized. Did these rules become understood through testing or is there more advanced documentation available that I just haven’t stumbled upon?

Also, what’s the significance of your bolded words? If you’re denoting importance, I understand why you bolded adjectives (‘second’). But what about words like ‘to’ or ‘by’ or ‘of all’? Are these just words/phrases that the model typically understands contextually?

2 Likes
6

No, the bolded is not mine - this is the forum markdown editor made for code. But the words in capital letters are mine, to get more focus from the model.

From now on please consider the following sentence: “In my humble opinion (IMHO),…”

The OpenAI documentation is a detailed technical description but it doesn’t contain all these “tips”. Most of these AI-grammatic rules came from the experimental results of the members of this community and from the models themselves, e.g. How is the best way to present you a list? or What is the best way to an OpenAi model understand an instruction?. It is cumulative knowledge - at this time we could consider this forum together with the OpenAI documentation as the most complete body of knowledge about AI-Language Models & Humans relationships - waiting for someone with patience enough to compile all the information.

For example:

The use of delimiters is the first tactic learned in the “Prompt Engineering” (free) training course, hence the use of punctuation advised to me by @EricGT here in the forum - while the itemization and number sequencing are advised by the models themselves.
Please notice in the last (instruction 6.) is terminated by a period “.”

The Large Language Models use Embeddings (multi-dimensional math vectors applied to words) mainly and tokenization as a measurement to understand the natural human (written) language under contextualization. I consider the best sources for embeddings:

  1. Word embedding - Wikipedia
  2. Cosine similarity - Wikipedia
  3. Embeddings - OpenAI
  4. @curt.kennedy - I read everything he writes about embeddings. His quote: Embeddings = Knowledge;

Prompt engineering best sources:

  1. ChatGPT Prompt Engineering for Developers - DeepLearning.AI
  2. Text completion - OpenAI Docs
  3. @ruby_coder - developed an integrated support system for OpenAI model users;
  4. @PaulBellow - the most active and experienced supporter in the forum;
  5. @EricGT - experienced developer with a strong detailed view of prompt engineering. He made a thread about ChatGPT prompting that I now consider mandatory: Helpful hints about using ChatGPT
  6. @ruv - maybe one of the best developers with extensive knowledge in Language Models integrated into programming languages. He made a very good topic, which I also consider mandatory: Cheat Sheet: Mastering Temperature and Top_p in ChatGPT API (a few tips and tricks on controlling the creativity/deterministic output of prompt responses.)
  7. @Luxcium - all-in-one challenging prompt engineering;
  8. ChatGPT itself
  9. Learn Prompting - advised by @EricGT;
  10. and many more.

What I was able to compile from the above, for which I am very grateful:

  1. Struggling with ChatGPT-3.5 and Seeking Help
  2. Force api response to be in non English language. How?
  3. Do you also have this problem or maybe you found the solution? Does OpenAI have official information about this situation. Is anyone is aware of OpenAI position on the topic?
  4. Surprising spelling and grammar issues → turned out a jailbreak vector
  5. Fake quotes being generated for summaries
  6. Random response appended at task completion - importance of delimiters and punctuation.

These sources are a very good beginning. I am sure you’ll enjoy a new view of AI language models.

5 Likes
7

Thanks.

You are giving me more credit than I deserve. At present my prompt engineering is more of an art than a science but as I have noted before my background with programming is different than most and leveraging that has proven beneficial. Specifically my knowledge of

comes in quite handy along with a BS in computer science over 40 years of programming experience. I typically create several parsers a year using DCGs for various problems, currently working on a DCG parser for GABC. (ref)

With regards to Prompt Engineering consider adding this site to your list

and read as many papers related to LLMs as you can. A good daily dose of

helps.

Specifically with regards to OpenAI see tokenizer page and keep an eye on ChatML which AFAIK is not official but when it is should be a game changer for prompting.

After I wrote this saw that @PaulBellow noted ChatML earlier, nice to know we think alike.

4 Likes
8

I created this yesterday for a similar line of thought in using ChatGPT to teach Prolog programming and wanted to see if ChatGPT responded as needed. The result was promising.

(Click triangle to expand)

Prompt
Grade this one sentence statement on a scale of 1 to 10 with 10 being best. Explain grade and how to make it better.

Prolog is a logic programming language that uses syntactic unification and the primary means of evaulation.

Completion

I would give this sentence a grade of 6 out of 10. Here’s why:

  1. The sentence contains a typographical error: “evaulation” should be “evaluation.”
  2. The phrase “the primary means of evaulation” is ambiguous and could be clarified.
  3. The sentence lacks a clear subject, making it difficult to understand who or what is using syntactic unification.

To improve the sentence, I would suggest the following changes:

  • Correct the typo by changing “evaulation” to “evaluation.”
  • Clarify the phrase “the primary means of evaluation” by specifying what is being evaluated.
  • Add a clear subject to the sentence.

Here’s a revised version of the sentence:

“Prolog is a logic programming language that uses syntactic unification as the primary means of evaluating queries and solving logical problems.”

This revised sentence corrects the typo, clarifies the purpose of syntactic unification, and adds a clear subject (“Prolog”). As a result, the sentence is easier to understand and conveys its intended meaning more effectively.

Prompt

Grade this one sentence statement on a scale of 1 to 10 with 10 being best. Explain grade and how to make it better.

“Prolog is a logic programming language that uses syntactic unification as the primary means of evaluating queries and solving logical problems.”

Completion

I would give this sentence a grade of 9 out of 10. Here’s why:

  1. The sentence is clear and concise, effectively conveying the key characteristics of Prolog as a logic programming language.
  2. It accurately describes the use of syntactic unification as the primary means of evaluating queries and solving logical problems in Prolog.
  3. The sentence is grammatically correct and free of typographical errors.

The only minor improvement I would suggest is to clarify that syntactic unification is used in the process of inference, which is central to Prolog’s operation. This clarification provides additional context for readers who may not be familiar with the concept of syntactic unification.

Here’s a revised version of the sentence:

“Prolog is a logic programming language that uses syntactic unification as the primary means of inference in evaluating queries and solving logical problems.”

This revised sentence retains the clarity and conciseness of the original while providing additional context about the role of syntactic unification in Prolog’s inference process. With this improvement, I would give the revised sentence a grade of 10 out of 10.

Prompt

Grade this one sentence statement on a scale of 1 to 10 with 10 being best. Explain grade and how to make it better.

“Prolog is a logic programming language that uses syntactic unification as the primary means of inference in evaluating queries and solving logical problems.”

Completion

I would give this sentence a grade of 10 out of 10. Here’s why:

  1. The sentence is clear, concise, and informative, effectively conveying the key characteristics of Prolog as a logic programming language.
  2. It accurately describes the use of syntactic unification as the primary means of inference in evaluating queries and solving logical problems in Prolog.
  3. The sentence is grammatically correct and free of typographical errors.
  4. It provides context about the role of syntactic unification in Prolog’s inference process, which is helpful for readers who may not be familiar with the concept.

Overall, the sentence is well-constructed and effectively communicates important information about Prolog. It is precise in its language and provides a good introduction to the subject matter. As such, I believe it deserves a grade of 10 out of 10.

Since the sentence is already well-written and informative, there are no significant improvements that I would suggest. However, it’s worth noting that the sentence is a standalone statement, and additional context or examples may be needed for readers who are not familiar with logic programming or Prolog. This additional information could be provided in subsequent sentences or paragraphs, depending on the context in which the sentence is used.

Funny side note: When I create prompts it often seems like the quality of the response, not the response itself, is like possible answers you get from a Magic 8 ball.

Also I find that when it comes to getting an LLM to do evaluations such as math or executing an algorithm that too often the result will be an hallucination. Instead the tatic I am currently choosing to pursue is to use ChatGPT plugins to do the evaluation and pass that back, this is a work in progress and yes I do have access to ChatGPT plugins but only just a few days ago.

Since Discourse will not let me post more than 2 replies in a row have to combine the following in this post.

As @AlexDeM notes, punctuation matters to an LLM.

Having created many parsers over the years, not to be confused with compilers which make use of parsers, one has to naturally think in terms of Abstract Syntax Trees and syntactic sugar. Many AST will use punctuation as operators (,,.,;, - … ) and these operators tend to be higher up in the tree. Many programmers do not think of the , at the end of a line in many programming languages as a binary operator but it is, without getting into details it often denotes the sequential execution of a statement. A period is even often higher and denotes the end of function. In a similar manner when creating prompts pay very close attention to the punctuation. As I noted to @AlexDeM before (ref) I have seen a period be interpreted as separating two commands when the intent of the second sentence was to add clarity to first. By changing the period to a comma the prompt worked as expected. @AlexDeM is the only one who I have seen to quickly embrace this knowledge and pass it along.

@AlexDeM Really liked your prompt here.

@james.earthman Notice that in that prompt numbers were used because there are 6 instructions but if instead they were not instructions to be executed sequentially then seriously consider using bullets * instead when creating list of clarification (constraint) statements.

One other item of note when creating prompts: Learning the preferred words.

While it may seem that my rules for creating prompts are nit-picky, keep this quote in mind as at times it overrules some of the rules.

“How well we communicate is determined not by how well we say things but how well we are understood.”

— Andy Grove

3 Likes