The topics ITT are better suited for tool calling. Here is a full working example that doesn’t require ANY system prompt.
import openai, json
from typing import Annotated
from tooldantic import ModelBuilder
client = openai.OpenAI()
sample_data = """
# Personal Health
- Weight is 75 kg {physical health, !2}
- Sleep quality is poor {well-being, !1}
- Daily steps are 4,000 {physical activity, !2}
- Stress level is high {mental health, !1}
- Water intake is 1 liter {hydration, !2}
- Meditation practice is 5 minutes daily {mental health, !3}
# Career
- Job satisfaction is low {professional growth, !1}
- Current project status is behind schedule {work performance, !2}
- Skill development is stagnant {career growth, !2}
- Networking efforts are good {professional relationships, !3}
- Work-life balance is poor {well-being, !1}
- Annual performance review is red {job security, !1}
"""
def health_effects(
positive_effects: Annotated[list[str], "List of all the bullet points in the text that yield **positive** effects on health."],
negative_effects: Annotated[list[str], "List of all the bullet points in the text that yield **negative** effects on health."],
):
"""Always use this tool to extract health data from the user supplied text. It will return the number of positive and negative effects found in the text."""
return json.dumps({
"number_of_positive_effects": len(positive_effects),
"number_of_negative_effects": len(negative_effects),
"message_to_assistant": "Please provide a summary of the health effects of the text. And report the numbers found in the text."
})
HealthEffectsModel = ModelBuilder().create_model_from_function(health_effects)
messages = [{"role": "user", "content": sample_data}]
r = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=[HealthEffectsModel.model_json_schema_openai()],
tool_choice='required',
parallel_tool_calls=False,
)
message = r.choices[0].message
messages.append(message)
tc = message.tool_calls[0]
args = json.loads(tc.function.arguments)
results = health_effects(**args)
messages.append({"role": "tool", "tool_call_id": tc.id, 'content': results})
r = client.chat.completions.create(
model = "gpt-4o-mini",
messages = messages,
)
print(r.choices[0].message.content)
# ### Summary of Health Effects
# #### Positive Effects:
# 1. **Meditation Practice**: Engaging in meditation for 5 minutes daily can contribute to improving mental health and stress management.
# 2. **Networking Efforts**: Good networking can enhance professional relationships and may lead to better career growth and opportunities.
# #### Negative Effects:
# 1. **Weight**: At 75 kg, there may be concerns regarding physical health, especially if it is higher than recommended for the individual's height.
# 2. **Sleep Quality**: Poor sleep quality can significantly affect overall well-being and cognitive function.
# 3. **Daily Steps**: Walking only 4,000 steps a day is below the recommended levels for physical activity, which can impact physical health.
# 4. **Stress Level**: A high stress level negatively affects mental health and overall quality of life.
# 5. **Water Intake**: Drinking only 1 liter of water may be inadequate for proper hydration.
# 6. **Job Satisfaction**: Low job satisfaction can lead to decreased motivation and well-being.
# 7. **Current Project Status**: Being behind schedule can increase stress levels and impact work performance.
# 8. **Skill Development**: Stagnation in skill development can hinder career advancement and professional growth.
# 9. **Work-Life Balance**: A poor work-life balance can lead to burnout and negatively impact mental health.
# 10. **Annual Performance Review**: Receiving a 'red' status in the annual performance review can affect job security and lead to increased stress.
# ### Counts Found:
# - **Number of Positive Effects**: 2
# - **Number of Negative Effects**: 10
Personally I see following possibilities to improve the results:
Properly passed input data so that you have a robust rag engine to find information within large documents and feed that information to your summarizing models.
Design the workflow to produce a single summary per smallest item
Design workflow to consume multiple summaries from the previous step and select the important information to generate the higher level summary
Introduce data collection for inputs outputs on the steps two and three to be able to find tune the models to further improve the results
If no intermediate analysis steps are required within your workflows the above would be pretty easy.
On the other hand if a deep analysis is required per item, in this case it might be tricky and you definitely need a specialist.
This also exactly what my data looks like! Only I have about 90 “valuelines”. It is also markdown, similar in format, etc. I actually don’t want a count; I just want the AI to list the valuelines with negative values (either based on color or other negative value) if it matches the subject too. That’s where the AI comes in. Example, question “what health items do I need to pay attention to?”. In your example, it might nothing nothing or things are pretty good but you might address stress. What it did before the extreme prompt engineering was miss a very obvious value like “red”.
Thanks for those suggestions Serge. On your points:
Yes in our app the UI allows the user to select the most relevant cut of the input/external data using filtering selections, before submitting the knowledge to the LLM.
Get your point but in terms of generating marketing content, aggregating multiple thoughts and concepts to produce coherent summary text for a particular use (e.g. web page, slides, product fact sheet) is often the objective (but I may have misunderstood you here).
Yes exactly, but selecting the most important information from something that’s been generated by GPT requires human processing, so users have to be trained in how to edit and sometimes overwrite prompts that have been suggested by the app.
We haven’t tried this yet but will take a look at it!
Thanks for that pointer, I took a look. I guess in our application of writing marketing content, we want a good degree of creativity from GPT (just as you would from a human writer). So that fact that GPT writes different text when you ask it to do exactly the same thing using exactly the same input knowledge is probably the behaviour we are looking for.
What I would say is while the precise wording is different each time, the concepts, arguments and value propositions that GPT chooses to feature in the summaries it writes are remarkably consistent.
Here is what was my approach (still in dev) on a related goal:
Take product description → condense/preprocess it to an analysis of the product → identify potential customers → find the base for the campaign → create campaign → fill campaign with ads.
Identify the final goals of the app (for me it was ads, but for you, it is more than just ads: social media posts, blog posts, etc.)
See how those are built (complete workflow in reverse order of how you get from the final result to initial input)
This will produce the list of initial inputs - those inputs will be the result of the step you need to do right before building the final output (we are here is reverse order) → take those inputs and go with them to step #2 to see how they are built
Iterate steps 2->3 to get the full workflow in reverse order from the final output to the initial input you originally insert in your app.
This will give you the picture of what your original input must look like after preprocessing/transforming it into the “workable” data (I think your current thread is more about this step
The above will more or less define your preprocessing logic and workflow
So as a result you will get:
Original input
How it should be preprocessed to make it usable by your app
Workflows to go from usable data to each final result (more like a bunch of paths to take the input to the final result)
Clear picture of your app logic and pitfalls
Good idea to introduce an extra step before the final output that would apply “tone skin” (brand voice, wording, SEO, etc) to the result before returning it to the user (fine-tuning is your friend here)
As for the workflow steps, hard to tell without details, but I’m sure good RAG and simple operations will help you get rid of most of the issues you have.
Feel free to reach out to me if you want to discuss it in more detail.
Thanks for this explanation of your dev project, which sounds very interesting.
BPM is a consulting firm that’s been doing go-to-market messaging work with blue chip clients for 20+ years, so we’re essentially approaching the LLM opportunity from the other end.
We have a methodology we’ve been using to build databases of insights and messages and a software platform to support this – Messaging Workbench. So we’ve hooked up GPT4o to one of these databases (API integration with the Workbench) and have been testing out the generation of product marketing content. When you get the prompt engineering just right, the output results are quite remarkable!