I have been messing with Davinci - instruct these days to summarize lengthy conversations between students and teachers. As a testing ground however I have it summarize conversations from chat applications. I will use this example to show the struggle:
katia: i finally got fridays money
katia: i just walked in and theyre like yo we thought you forgot about it
katia: ill miss big G the manager so much
katia: one of my favorite people
Phil: work no more?
Phil: congrats đź‘Ź
Alex: exactky
katia: its been like 3 weeks
Phil: ah\
katia: i considered just never going back bc seeing everyone would be so awkward
katia: maybe i just worked as charity work
Phil: So you would rather avoid the awkward meeting than getting paid
Phil: lmao
Phil: didn't they call u at all
katia: no
katia: they said they would
katia: also every week i told them to send me the schedule over email
katia: and theyre like ye ye sure we’ll do it
katia: and they never did so i had to call them every sunday
katia: and then on my last day im like yo what the fuck why did u never send me the schedule
katia: and hes like we dont do that lol
katia: company policy my ass
Borris: Cyprus
Phil: ^
Phil: !tldr 50
The way I structure the prompt is as following:
Convert the following conversation between users into a summary:
{conversation}
Summary:
With the settings as follows:
Now the issue is that the AI is incapable of summarizing everything, or not making up things. For example:
Attempt 1
Summary:
Katia was hired to work for Fridays in Cyprus. When she asked for the schedule, they said they didn’t provide it. She asked for the schedule every Sunday, but they never sent it. When she finally quit, she found out that they were lying about company policy.
The above attempt although good, it misses the initial part of the conversation.
Attempt 2
Summary:
Katia found out her managers had been lying to her and wanted to quit because she didn’t feel like she could go back. She was considering working at the store as a volunteer.
This is takes too many guesses.
Maybe this was a bad example. For another more lengthy example, the AI tends to freestyle when it comes to the responses such as producing bullet points:
Summary:
Gabriella says they weren’t looking through the messages but they did look at the book when they were handed it by Marina.
Marina says she was drunk and handed her phone to Gabriella.
Max says Marina wasn’t drunk.
Gabriella says they never took advantage of Marina.
or generates additional information out of nowhere:
Marina was not drunk when she showed Gabriella the book, but she wasn’t sober either.
Marina showed the book to Gabriella because she was curious about it.
Marina’s friends can attest to the fact that Marina was not drunk when she showed Gabriella the book.
Marina was aware that the book was about old drama.
Gabriella never took advantage of Marina because she was drunk.
( from this point onward this is false)
Gabriella got the book from Marina as a gift.
Gabriella never asked to see the server during the outing before Marina showed her the book.
Marina was the one who mentioned the book at the outing.
The server was deleted because it was filled with old drama.
Marina mentioned the book at the outing because she didn’t know that everyone didn’t know
Let me know what suggestions you might have I can experiment on for more sturdy results.
The devil is in the details. Or, more specifically, the devil is your lack of adjectives. You have to be more pacific about what you want the output to look like by using descriptive words.
Write a detailed summary paragraph of the following conversation:
katia: i finally got fridays money
katia: i just walked in and theyre like yo we thought you forgot about it
katia: ill miss big G the manager so much
katia: one of my favorite people
Phil: work no more?
Phil: congrats
Alex: exactky
katia: its been like 3 weeks
Phil: ah
katia: i considered just never going back bc seeing everyone would be so awkward
katia: maybe i just worked as charity work
Phil: So you would rather avoid the awkward meeting than getting paid
Phil: lmao
Phil: didn’t they call u at all
katia: no
katia: they said they would
katia: also every week i told them to send me the schedule over email
katia: and theyre like ye ye sure we’ll do it
katia: and they never did so i had to call them every sunday
katia: and then on my last day im like yo what the fuck why did u never send me the schedule
katia: and hes like we dont do that lol
katia: company policy my ass
Borris: Cyprus
Phil: ^
Phil: !tldr 50
Paragraph summary:
Katia is discussing her last day of work and how she’ll miss her co-workers. Katia mentions that it had been 3 weeks since she quit and had been considering never going back since it would be awkward. Phil suggests that she might as well not go back and work as charity work. Katia says she is not sure if she wants to do that and Phil says she would rather avoid the awkward meeting than getting paid. Phil then suggests that they never called her and that she had to call them every Sunday and then on her last day she found out that they don’t email the schedule. Katia says that they never emailed the schedule and that company policy is a lie.
Write a detailed bullet point summary of the following conversation:
katia: i finally got fridays money
katia: i just walked in and theyre like yo we thought you forgot about it
katia: ill miss big G the manager so much
katia: one of my favorite people
Phil: work no more?
Phil: congrats
Alex: exactky
katia: its been like 3 weeks
Phil: ah
katia: i considered just never going back bc seeing everyone would be so awkward
katia: maybe i just worked as charity work
Phil: So you would rather avoid the awkward meeting than getting paid
Phil: lmao
Phil: didn’t they call u at all
katia: no
katia: they said they would
katia: also every week i told them to send me the schedule over email
katia: and theyre like ye ye sure we’ll do it
katia: and they never did so i had to call them every sunday
katia: and then on my last day im like yo what the fuck why did u never send me the schedule
katia: and hes like we dont do that lol
katia: company policy my ass
Borris: Cyprus
Phil: ^
Phil: !tldr 50
Bullet point summary:
-Katia is talking about how she got paid for her last day of work, which she is happy about.
-She feels bad for her manager, who she liked a lot.
-She has been considering not going back to work because it would be awkward to see the people she worked with.
-Phil asks her if she would rather avoid the awkward meeting than get paid, and she says that she considered just never going back to work.
-Katia says that she wanted to get the schedule over email, but that the manager told her that they don’t do that.
-The manager also never sent the schedule over email.
-Katia mentions that she called the manager every Sunday to get the schedule, but that he never sent it over.
-Katia has been working as a charity worker for 3 weeks now.
-Phil says that she should go back to work because she was there for 3 weeks.
-Katia asks if she can work for charity.
-Phil says that she should go back to work because she has been there for 3 weeks.
-Phil says that she should work no more because she has been working for charity for 3 weeks.
-Alex agrees.
(Also we default settings). Better use of the English language is far more important than hyperparameters here.
