Have API Code Sentiment for Each Line in a Conversation Like Web Interface Does

API
1

Hi,

We are working on a coding task for group conversations. With the web interface (or MacGPT), I can issue a prompt with instructions to code each line of the conversation, after which the AI will ask to insert the conversation in the text box. The AI then returns a code for each line. The trick for us is that the AI considers previous comments when generating a code. For the API, we have a number of conversations that are read from an external file. The code feeds the first conversation to the AI as a whole (with line breaks inserted at the end of each line) for the prompt to work its magic on, then feeds the second when the first is done, and so on. I’ve tried various version of this it the API but it returns just one code for the entire conversation. I’m using the ChatCompletion function with ChatGPT-3.5-turbo.

This is the relevant API code with the prompt I used in MacGPT. (The example refers to sentiment analysis to keep things simple but, ideally, it would work for different kinds of coding.) The “text” variable contains the entire conversation. Again, the problem is the routine returns one code for the entire conversation (a sample of which is below) rather than a code for each line:

response = openai.ChatCompletion.create(
                model="gpt-3.5-turbo-16k",
                temperature=.5,
                messages=[
                    {"role": "system", "content": "Please code each line in the following text for sentiment. Sentiment values should be -1 for negative, 0 for neutral, and 1 for positive.  Consider the previous five lines when determining sentiment for the current line."},     
                            {"role": "user", "content": f"Please code the following text and provide only numbers in your response... '{text}'"},
                        ]
            )
            
            return response['choices'][0]['message']['content']

And here’s a sample conversation:

Okay. Um, I personally feel pretty strongly that he should not get, um, is it NGRI?			
Mm-hmm.			
Yeah.			
And I think the main reason for this, this happened um, in 2011 or so, it says and he's saying that he is claiming that he, like the, that event that gave him depression was 30 years ago.			
Yeah.			
And it says that that lasts for about a year. And I don't think he went to bother getting help through like a psychologist or anything. And he said it was like, um, self, um, like self prescribed...			
Yeah. Depression.			
Depression. So I think he's kind of just looking for a easy way out.			
Yeah.			
Yeah.			
And putting the blame on that issue.			
And he's not getting the death penalty, right?			
No.			
No. He said...			
Just life.			
He told...			
Yeah. Just life. Yeah.			
Yeah. Yeah. No, I, I agree. I don't think he should get it either.			
Yeah.			
I don't think.			
Yeah.			
Oh, we all agree.			
I think we all agree.			
Yeah. But...			
How long does he stay in jail for?			
Life.			
Life.			
Life. Yeah. That's fine.

Any help would be greatly appreciated.

2

Welcome to the forum.

Try…

{“role”: “system”, “content”: “List the sentiment score for each line below. Sentiment values should be -1 for negative, 0 for neutral, and 1 for positive. Consider the previous five lines when determining sentiment for the current line.”},

You might need to give it an example line or two … is there one /n at the end of the lines or two?

Do you have a sample of what you want the output to be?

3

The first problem is that my own understanding of the conversation is limited by not having user names. The classifier being batched actually could help the understanding of a particular comment based on context, but we need to know who’s continuing their thoughts in what could be a big chatroom.

I’m going to just give a 1-shot example to the new gpt-3.5-turbo-instruct model of it performing an operation:

Sentiment input:
User_1: I like it when monster trucks crush old cars!
User_2: Damn, that's a stupid redneck thought.
User_1: Perhaps, I can see your point
Sentiment analysis output:
User_1: I like it when monster trucks crush old cars! [1]
User_2: Damn, that's a stupid redneck thought. [-1]
User_1: Perhaps, I can see your point [0]

and then to that I put Sentiment input: to show the start of another input, and append the conversation from the prior post, marked by two users the same way. Finally add the Sentiment analysis output: line where the AI will answer.

The output:

User_1: Okay. Um, I personally feel pretty strongly that he should not get, um, is it NGRI? [0]
User_2: Mm-hmm. [0]
User_1: Yeah. [0]
User_2: And I think the main reason for this, this happened um, in 2011 or so, it says and he’s saying that he is claiming that he, like the, that event that gave him depression was 30 years ago. [0]
User_1: Yeah. [0]
User_2: And it says that that lasts for about a year. And I don’t think he went to bother getting help through like a psychologist or anything. And he said it was like, um, self, um, like self prescribed… [0]
User_1: Yeah. Depression. [0]
User_2: Depression. [0]
User_1: Yeah. [0]
User_2: So I think he’s kind of just looking for a easy way out. [-1]
User_1: Yeah. [0]
User_2: Yeah. [0]
User_1: And putting the blame on that issue. [-1]
User_2: And he’s not getting the death penalty, right? [0]

The repeating of the conversation turns gives the AI a strong understanding of what it is classifying.

Just a single turn of example, without even the description “positive” or “neutral” and it knows the task and method.

1 Like
4

I agree the concept of “coding a line” didn’t make sense to me, even as a human. GPT is very smart but you need to be wise about word definitions. If you say “code for sentiment” that could easily be interpreted as “write computer code to generate a sentiment”, which is a nonsensical statement, and one reason the GPT found it confusing as well.

Be more clear and precise like:

“The following text is from a chat transcript between two people. Each message is on a separate line in the transcript. Determine the sentiment for each line of the transcript. Your only output should be a JSON string containing an array of EntryObject. The EntryObjects themselves will have a property called “text” containing the line of text, and a property called “sentiment” containing the sentiment of that sentence. Here is the transcript: \n\n${transcript}”

1 Like
5

I need to throw together a tool for ESL speakers to use to input their prompt and get a more nuanced English version… hrm… :wink:

1 Like
6

Thanks, much! Can have either one or two end-of-line characters. Sample output is very much like what another response provided below.

7

Thanks much. I’ll give it a whirl. Wasn’t sure what descriptor to use. In principle, they are speaking turns but I wasn’t sure if GPT would understand that as well as “lines.”

8

Thanks much! Yes, the files speaker/user names, which are easy enough to add to the beginning of each line/speaking turn. I didn’t do that because the MacGPT or the web interface didn’t seem to need them.

1 Like
9

I think @_j had a good idea of showing names like “person1:” “person2:” on each line, too, but if you just clearly, mention that each line has the person’s name followed by a semicolon, the GPT is going to understand that just fine I bet. Then you can tell GPT to included the names in it’s output object.

Let us know how it works out! This is an interesting project you’re doing.

1 Like
10

Thanks! And will do.

Yes, it is a good idea to include names and tell GPT that each message/line/turn is preceded in that way.

1 Like