I have an odd use case I’m trying to apply LLMs to solve. My datasets consist of long chat logs between numerous individuals and I’m attempting to quantify the emotional state of the collective group. These logs can be big (hundreds of messages, 100k+ tokens). Example queries (simplified):
How upbeat is the collective chat? 0 = universally poor mood, 100 = everybody upbeat.
What is the socioeconomic status of the chat participants? 0 = of lower means universally, 100 = all participants are economically elite.
Is there a perceived sense of fairness at economic circumstances of chat participants? 0 = universally UNequitable and UNfair, 100 = totally fair.
Are users drawing a sense of belonging and inspiration from the chat?
and so on.
I’m struggling to get ChatGPT 4o or 4o-mini to make a critical assessment. More often than not, my ratings are wishy washy, bland, middle of the road judgements. I want some clear signal from the noise here but struggling.
My prompts consist of:
Pretty descriptive system prompt.
The full chat log with username, timestamp, and text content.
A JSON-structured output response format with field descriptors, well defined fields (int, str, etc).
Basically I’m just frustrated with how lacking in meaningful signal I’m seeing. I’m wondering if theres a better way to (1) handle the chat log inputs to ChatGPT, and (2) define the emotional field criteria better to get more meaningful or signal-rich content.
Without training on the parameters of what they are it’s very vague for such a large data set. You need parameters for it to gage from. It would be hard to analyze that much emotional context with no direction to graph it. Without focus it will always try to spin it positive. You have to use be critical in the prompt and set ranges in catagories for what you are looking for.
Keywords: Look for context-specific keywords that signal emotional shifts (e.g., “happy,” “angry,” “disappointed”).
Phrases: Detect sentiment in conversational phrases like “I feel” or “It seems.”
Graph Emotional Trends Over Time:
Use timestamps to graph emotion intensity over conversation sections.
Highlight trends in emotional peaks and troughs.
Set a Critique Bias:
Encourage the model to be analytical rather than optimistic. For instance, if someone expresses frustration, don’t downplay it; instead, focus on exploring the reasons and categorizing the intensity.
Critical Sentiment Filtering:
Filter for Neutrality: Ask the model to avoid unnecessary positivity and focus on analyzing both the negative and positive aspects equally.
Prompt Example:vbnet
Copy code
Analyze this chat log for emotional content. Identify emotions within the categories of Positive, Negative, and Neutral. Use a scale from 1 to 5 to rate the intensity of each emotion, and ensure critical neutrality by not biasing responses toward positivity. Categorize complex emotional combinations as distinct from simple positive or negative feelings. Graph the trends over time.
Output Categories:
Emotion Count: How often each emotion is expressed.
Top Intensity: Highlight the moments of highest emotional impact.
Overall Sentiment: Provide an average score for the conversation’s emotional tone.
By critically defining parameters and using ranges, you give the model direction, avoiding vague positivity and ensuring a balanced emotional analysis.
Do you think I should include this index guide on a per-emotion basis? Or is this better left as a system-level prompt?
I make frequent use of structured outputs to keep the outputs consistent in nature. Related to point #1, do you think this “instruction guide” could be better placed in the description field of the structured output pydantic framework?
In general I struggle with where the right prompts go: do I use the system/user/assistant fields versus inserting prompting guides in the description fields of the structured outputs? I haven’t seen many great examples to help clarify for me.
You could set it up as a command if used in a GPT instruction, If your prompt is set as an instruction you can give it a tool command by naming it. Like name it demographer or something. list the instructions with it add command like “ demographer instructions” add when user types emo use demographer instructions for user prompt query. So you could say use emo scan chat for sadness and rank it as to your graph. There is no wrong way to activate code emo. Just say use emo and add parameters to query. The logic could be set up many ways depending on your needs.
Here’s a more detailed breakdown:
Naming the Command:
Name the command something like “demographer” to represent its purpose (tracking or analyzing emotions). Once named, this command can be triggered when a user wants to analyze or categorize emotions in their prompts.
How to Implement:
Add instructions tied to this command. For example, you could write “Demographer Instructions” and include the logic that should execute when the user types in an emotion-related query. The system can then automatically run these instructions when certain emotional keywords (like “emo” for emotion) are used.
Setting Up the Scan:
If a user types something like “emo scan,” you can program the system to check the conversation for specific emotions (e.g., sadness, happiness). You could then rank or graph the emotions based on frequency or intensity. For example:
If the user types “emo scan for sadness,” the command will search through the chat and detect sadness-related cues, providing an analysis or ranking.
Flexibility in Activation:
There’s no single “right” way to set this up. You could customize it based on how you want users to interact with emotions in the system. Whether it’s analyzing the emotional tone of a conversation or querying a specific emotion, the logic can be tailored depending on the use case.
Parameters for Customization:
Users could adjust parameters—like setting a threshold for when an emotion should trigger a response or choosing specific emotions to analyze. This makes the tool adaptable to various needs, whether it’s for emotional monitoring or user interaction analysis.
Example of instruction logic.
How to Set Up a Command for Emotional Data Analysis in GPT Using Tool Logic
Define the Command Name:
Start by naming the command. You can name it something like “Demographer” or “EmoAnalysis”, based on your task. This will be the identifier for the command that runs your emotional data analysis.
Set Up Tool Command Logic:
You can define instructions that should execute whenever a user runs this command. For example:
text
Copy code
Tool Command: DemographerInstructions
This command can now be used whenever someone needs to analyze emotional content in a chat.
Activate Based on Emo Scan:
When the user wants to analyze the emotional content, they can type a phrase like:
text
Copy code
emo scan chat for sadness
The system will then trigger the “DemographerInstructions” to run an emotional scan of the chat for sadness, rank its intensity, and provide a graph showing emotional trends.
Include Parameters for Customization:
You can make the command flexible by adding different parameters for the user to adjust based on their needs. For example, the user could specify:
Emotion Type (e.g., sadness, joy, frustration)
Intensity Range (e.g., low, medium, high)
Graph Output (e.g., to display a visual representation of emotional trends)
This ensures the command can be tailored for different emotional queries.
Example Tool Command Logic:
Here’s how the command logic might look:
text
Copy code
Tool Command: DemographerInstructions
Use emo scan for emotions like sadness, joy, or frustration.
Rank emotional intensity as low, medium, or high.
Graph emotional trends over time and display peaks.
In this case, whenever someone triggers the “DemographerInstructions” tool with an input like “emo scan chat for sadness,” it will follow these instructions to scan for the specified emotion, rank the intensity, and provide a graphical output.
Versatility of the Command:
There’s flexibility in how you set it up. The user could type commands like:
text
Copy code
emo scan for frustration and graph it
or
text
Copy code
emo scan chat, rank anger from 1-5
Each variation of the command will invoke your defined tool logic, but allow for different queries based on the user’s input.
Conclusion:
The logic is modular, meaning it can be adapted for a variety of emotional analysis tasks. There is no wrong way to set it up—just ensure that the core structure follows the sequence of: