Role:assistant vs role:user - major differences?

Prompting
1

I m using chatGPT to extract content from text and create questions from the text.

Recently I was struggling to get good results in terms of the output.

  1. I need to have an array of objects, each with a question and answer → Most of the time I was getting a single object back instead of an array.
  2. My prompts are quite complex and heavily customized → inconsistent responses - for example if I ask for the results in a different language than the original language the api only complies about 50% of the time.
  3. Inconsistent formatting. → I would get responses like {“responses”:[… or {“questions”:[ or ```JSON …

I tried many things to ensure better and more consistent responses. I was using 3.5 turbo originally.

I tried GPT4 preview → results were more or less than same as 3.5
GPT4 → better responses but its too expensive for my use case
Fine tuned GPT3.5 → surprisingly this gave me WORSE results. I fine tuned it with responses that were arrays of objects, always containing multiple objects, and yet the responses I got were always a single object.

I also played around with asking for the response to be in json_format, this somewhat works but again often ended up in the situation where I only had one item instead of a whole array.

I also couldn’t figure out why when using the playground I was getting great results compared to the API. Well it turns out that in the API I was feeding my prompt in as a message with the role “user”. In the playground the role is “assistant”. After changing it in my code to “assistant” the responses seem much better. (that said I went back to the playground and tried sending the content with “user” and the results good)

I need to keep testing but it has anybody had this experience? I dont understand why it would be the case. From the documentation I can’t see what impact the role would have here.

1 Like
2

Have the same question similar to yours in terms of role name in using API.

Human/User
Assistant/System
Message/Question/Answer

Hope there’s experienced experts to help us figure out what is behind…

3

they are not roles as in a users permission is admin. it’s roles of the message sent in order to use some as requests and others as context.
user system is to set behavior rules in the gpt asking to act different. can be used by he chat too btw.
assistant is providing query to answer and should be looked at as how the massager is acting if the chat is asked to retrieve information from the user it will act assistant
message is responding with a result

4

@wiseduckling
A beginning of answer is to read 2404.05624 paper on arxXiv
This paper submitted to arXiv on yesterday April 8th '24 try to test and compare different roles settings, tag combinations and prompts to tag named entities in sentences using GPT 3.5 API.
It use metrics (P, R, F1 ) and systematic benchmarking with datasets. Source code of the study is available on Github.

Conclusions of the paper about Named entities tagging:

  1. Named enties tagging is not a Chat task, it’s a text augmentation task => in the API call, put everything in role:assistant (AAA in the paper)
  2. More shots give better results=>30 shots!
  3. Some tag combinations for the output are better=> authors obtain better results with ##Entry##Label (eg ##FRANCE##LOC)

Others LLMs API could be better for some extraction but not for others. So, it’s recommanded to test it for each extraction workflow and uses cases with your own test dataset