I have an odd use case I’m trying to apply LLMs to solve. My datasets consist of long chat logs between numerous individuals and I’m attempting to quantify the emotional state of the collective group. These logs can be big (hundreds of messages, 100k+ tokens). Example queries (simplified):
How upbeat is the collective chat? 0 = universally poor mood, 100 = everybody upbeat.
What is the socioeconomic status of the chat participants? 0 = of lower means universally, 100 = all participants are economically elite.
Is there a perceived sense of fairness at economic circumstances of chat participants? 0 = universally UNequitable and UNfair, 100 = totally fair.
Are users drawing a sense of belonging and inspiration from the chat?
and so on.
I’m struggling to get ChatGPT 4o or 4o-mini to make a critical assessment. More often than not, my ratings are wishy washy, bland, middle of the road judgements. I want some clear signal from the noise here but struggling.
My prompts consist of:
Pretty descriptive system prompt.
The full chat log with username, timestamp, and text content.
A JSON-structured output response format with field descriptors, well defined fields (int, str, etc).
Basically I’m just frustrated with how lacking in meaningful signal I’m seeing. I’m wondering if theres a better way to (1) handle the chat log inputs to ChatGPT, and (2) define the emotional field criteria better to get more meaningful or signal-rich content.
Without training on the parameters of what they are it’s very vague for such a large data set. You need parameters for it to gage from. It would be hard to analyze that much emotional context with no direction to graph it. Without focus it will always try to spin it positive. You have to use be critical in the prompt and set ranges in catagories for what you are looking for.
Keywords: Look for context-specific keywords that signal emotional shifts (e.g., “happy,” “angry,” “disappointed”).
Phrases: Detect sentiment in conversational phrases like “I feel” or “It seems.”
Graph Emotional Trends Over Time:
Use timestamps to graph emotion intensity over conversation sections.
Highlight trends in emotional peaks and troughs.
Set a Critique Bias:
Encourage the model to be analytical rather than optimistic. For instance, if someone expresses frustration, don’t downplay it; instead, focus on exploring the reasons and categorizing the intensity.
Critical Sentiment Filtering:
Filter for Neutrality: Ask the model to avoid unnecessary positivity and focus on analyzing both the negative and positive aspects equally.
Prompt Example:vbnet
Copy code
Analyze this chat log for emotional content. Identify emotions within the categories of Positive, Negative, and Neutral. Use a scale from 1 to 5 to rate the intensity of each emotion, and ensure critical neutrality by not biasing responses toward positivity. Categorize complex emotional combinations as distinct from simple positive or negative feelings. Graph the trends over time.
Output Categories:
Emotion Count: How often each emotion is expressed.
Top Intensity: Highlight the moments of highest emotional impact.
Overall Sentiment: Provide an average score for the conversation’s emotional tone.
By critically defining parameters and using ranges, you give the model direction, avoiding vague positivity and ensuring a balanced emotional analysis.
