I want to write articles from interview transcripts.
I have a very clear idea of the article construction. I am a journalist who has done this across thousands of pieces. The article purpose is to showcase the interviewee’s views, and the role of the writer is to frame those with smart commentary or industry context.
The interview is always of an executive, by an interviewer. The AI can successfully differentiate. The industry and broad topic are always the same.
Ultimately, I want to chain together a workflow with an API for regular writing, and maybe text-davinci-003 would work for that. Until then, I’m using ChatGPT.
By and large, results are okay. However, encouraging OpenAI to behave as I would is proving a little tricky.
What should I improve or learn about this prompt? What do I need to learn?
Issues I am finding…
The AI is too keen to take material from the interview with which to make assertions of its own. The job of the journalist is to be more distanced than this.
It should not echo the interviewee’s views. I keep instructing it not to pass off the interviewee’s insights as its own. It’s paraphrasing/attribution often includes the same words as the interviewee. I keep trying to force it to use alternative wordings. The instruction to “paraphrase” should surely be enough.
I wonder if this is a conflict between two instructions. After all, I am also instructing it to reflect the views of the interviewee.
It is often ignoring my pleas not to use a direct quote in the first sentence, and also my request that a specific formulation of named attribution only happen in the sentence prior to the first sub-heading. Again, I wonder if the twin instructions “reflect the views of the interviewee” and “don’t make assertions of your own” leave the AI thinking, “Well what on earth do you want me to write in the first sentence then?”
Whilst, in isolated, slimmed-down tests, I have proven how ChatGPT can remove filler words / discourse markers / speech stylings / hesitations / stumbles from text, in this longer prompt, this is sometimes overlooked.
There is insufficient character in the writing - no smarts, few turns of phrase. Is this because asking it to reflect the views of the interviewee - and make no assertions, whilst not repeating in the paraphrase - is constraining? It wouldn’t be to a human.
I have to actively discourage the AI from putting its typical “in conclusion” or “in summary” at the end. This isn’t an academic essay. That’s fine. But sometimes it doesn’t listen.
Sometimes, the article doesn’t come in as long as I instructed.
With all my instructions plus a transcript of an >8-minute conversation, my full prompt is circa 1,295 words, so about 1,619 tokens.
This is a variant, though I keep trying to tweak it…
The following text is a transcript of an interview. Generate an outline of the three key points conveyed by the dominant speaker, ie the interviewee, but don’t show this.
Write a 650-word article as per this outline.
The article’s focus should be communicating the views and insights of the interviewee.
The audience for the article is xxxx xxxxx xxxx professionals.
Quotes:
The interviewee is Johnny Doe, SVP Widgets & Sprockets, Acme Inc.
66% of the article should be direct quotes. Attribute quotes in present-tense, eg. “says”, “thinks”, “claims”, “believes” - not "said, “claimed” etc.
When you attribute or summarise the speaker’s quotes or views in your own voice, you must use alternative words or phrasing. Do not just repeat material from a quote near-verbatim in your own writing. To avoid repetition, you are allowed to be more creative and varied in the language you use.
Sections:
Use sub-headings at the appropriate points, in line with the outline you produced earlier. Every sub-head should be prefixed with ## .
Break into a new paragraph approximately every two sentences.
Intro:
The first sentence should be a sassy, smart journalistic intro.
The sentence immediately before the first sub-head should segue to the section denoted by the first sub-head by using an attribution like this… “…xyz,” says [surname], in this interview with Publication Name." But only mention “Publication Name” once. Do not mention “Publication Name” in the first sentence.
Outro:
The end of the article should not be a summary or conclusion. You should not repeat material in the article. Instead, it should include forward-looking material.
Objectivity:
The article should be neutral. You must not positively endorse the company or interviewee. For example, do not write that the company or any category of company is “well-placed”, “well-positioned” or similar. Do not ignore this.
Do not endorse, echo or repeat the speaker’s sentiments. Your own writing should be neutral and not opinionated. You should always attribute the speaker. Do not make any assertions of your own. For example, where you paraphrase the speaker, state that he or she “says”, “thinks”, “claims” or “reckons”, or use “according to”.
Headline:
Use a smart main headline for the article. The headline should encapsulate the interviewee’s best view on the article’s main topic. Prefix it with # .
Flow:
Remove speech stylings and stumbles like hesitations or false starts, but do not distort the speakers’ quotes.
Hi
I can only talk from limited experience. Here are my observations.
ChatGPT is superior to the best model available via the API; text-davinci-003.
Knowing that I would test using the playground feature of the OpenAI website which use text-davinci-003.
I have found this model is nowhere near as good as ChatGPT when dealing with multi step request. For example. I can ask it to summarize text and if works well but if I ask it to “Summarize the text and if formatting is needed then and only then use markdown headings (#), ordered lists (1.), and unordered lists (-), while keeping the same case for headings and preserving the original content.” I get different output because it decides to give me output that has headings and lists even though they would not have previously been necessary.
Yes, it seems like text-davinci-003 does not cope well with the idea of “summarise the key points first, and then write an article based on that”, whereas ChatGPT does better. text-davinci-003 just spits out the key-points outline despite being told not to.
That said, this instruction may be my human brain’s linear thinking. It may be that this one-two punch is overly prescriptive, and that the model/s is perfectly able to write an article that would be segmented in about four thematic chunks, if only because I’m asking it to use sub-heads or because it would be good practice anyway. Any thoughts?
Ultimately, I don’t want to have a conversation to get to the article. I just want to provide a sure-fire recipe of instructions that works every time, automatically. And ultimately, I’ll want to do this via API request - sounds as if I may need to rely on the forthcoming ChatGPT API for that, if text-davinci-003 cannot handle it. Or does anyone here think I can pull this off using text-davinci-003??
Is that second formulation something you find works well, all the time? Are you doing this successfully with text-davinci-003 or with ChatGPT?
In my case, I will always want my articles broken up by ## sub-headings, about three/four of them, and then around three/four sentences for each chunk. It’s not an “if” this is required…
And it’s about more than “formatting”. The chunks between the sub-heads are essentially distinct thematic parts of the story. Can the model act accordingly just from being instructed to use cross-heads, or does it need to be told to find the key points before acting on anything?
Very interesting project. If you would like to brainstorm and experiment some more on it, as a fellow journalist, I’d be interested to contribute in any way possible.
You have asked a number of questions. I won’t address them directly, but here are some ideas that may help.
With Davinci, I have found (today) that if my input includes Markdown, I don’t have to ask it to format the output in Markdown. It intuits that this is what is expected in the output. So now I either rely on my text having it, or in the case of my summary question I hint at it. For example: “Create a summary of the text under the heading # Summary”.
Which leads me to another point. Examples in input lead to the best results. No example is called zero-shot, an example is one-shot and so on. Basically, your examples are teaching what you want, which is as good if not better than telling what you want.
Another thing to consider are pipelines. I have done this when the text is too long to input in one go. I took someone’s very long CV and fed in each section, one by one, that covered experience and had the AI summarise it looking for specific highlights (e.g. industry experience, etc.). I then fed all the summaries in together and had it summarise the summaries, asking to look again for specific highlights. I did this manually and it was quick. I got it down to one page in a few minutes. This would normally require reading and digesting the entire CV with notes and lots of back and forth.
I’ve experimented a bit on your prompt. In my experience, the prompt needs to be as clear and simple as possible. This is the result I got so far. I think this could start be used as a starting block and continue tweaking it further, as long as you’re clear about where to put what.
I’ve found that GPT-3 is not very good at completing multiple, nuanced tasks. Perhaps, it’s possible to them in multiple steps. For example, write the body of the article, and then later ask it to think of intro and titles etc…
TRANSCRIPTION:
OPRAH: We can’t hug, everybody is double- masked and has face shields. You look lovely. Do you know if you’re having a boy or a girl?
Meghan: We do this time. I’ll wait for my husband to join us and we can share that with you.
Oprah: That would be really great. Before we get into to it, I just want to make clear to everybody that, even though we’re neighbours, I’m down the road, you’re up the road, we’re using a friend’s place. There has not been an agreement, you don’t know what I’m going to ask, there is no subject that’s off limits and you are not getting paid for this interview.
Meghan: All of that’s correct.
Oprah: I remember sitting in the chapel — thanks for inviting me, by the way. I so recall this sense of magic. I never experienced anything like it. When you came through that door, you seemed like you were floating down the aisle. Were you even inside your body at that time?
Meghan: I’ve thought about this a lot. It was like having an out-of- body experience I was very present for. The night before, I slept through the night entirely, which is a bit of a miracle, and then woke up and started listening to Going To The Chapel, to make it fun and light and remind ourselves this was our day. We were both aware in advance of that this wasn’t our day, this was the day planned for the world.
Meghan: No. I didn’t feel any need to, because everything I needed to know he was sharing with me. Everything we thought I needed to know, he was telling me.
Oprah: So, you didn’t have a conversation with yourself, or talk to your friends about what it would be like to marry a prince, who is Harry, who you had fallen in love with . . . you didn’t give it a lot of thought?
Meghan: No. We thought a lot about what we thought it might be. I didn’t fully understand what the job was: What does it mean to be a working royal? What do you do? What does that mean? He and I were very aligned on our cause- driven work, that was part of our initial connection. But there was no way to understand what the day-to- day was going to be like, and it’s so different because I didn’t romanticise any element of it. But I think, as Americans especially, what you do know about the royals is what you read in fairytales, and you think is what you know about the royals. It’s easy to have an image that is so far from reality, and that’s what was so tricky over those past few years, when the perception and the reality are two different things and you’re being judged on the perception but you’re living the reality of it. There’s a complete misalignment and there’s no way to explain that to people.
ARTICLE:
Having experienced a combination of privilege and scrutiny unlike any other, Harry and Rachel have faced unique challenges that many have been unable to understand.
“There’s a complete misalignment and there’s no way to explain that to people,” says Harry. As an American, Rachel was able to bring a fresh perspective that allowed her to understand the differences between fairy tales and reality. “What you do know about the royals is what you read in fairytales, and you think is what you know about the royals. It’s easy to have an image that is so far from reality, and that’s what was so tricky over those past few years, when the perception and the reality are two different things,” she explains.
The couple understand the Royal Family better than anyone else and acknowledge that the way their lives are perceived is not always true to their reality. Rachel took time to consciously process the division between fairytales and her tangible experiences so that she could confront the judgments she was receiving about her life and marriage. “I didn’t fully understand what the job was: What does it mean to be a working royal? What do you do? What does that mean?” she asks.
Harry and Rachel’s decision to be intentionally present for their wedding was part of their plan to take ownership of the day.
“It was like having an out-of-body experience,” shares Rachel. “I was very present for. The night before I slept through the night entirely, which is a bit of a miracle, and then woke up and started listening to Going To The Chapel, to make it fun and light and remind ourselves this was our day. We were both aware in advance of that this wasn’t our day, this was the day planned for the world.”
Harry and Rachel have seen the world in unprecedented ways in their not-so-ordinary life. As they navigate their collective experiences together, certain moments have stayed with them, defining their relationship. “I recall this sense of magic. I never experienced anything like it. When you came through that door, you seemed like you were floating down the aisle,” recalls Rachel.
The Royal family welcomed Rachel in their modern world, with the intention of making her feel comfortable in her new environment. “We thought a lot about what we thought it might be,” says Harry. “He and I were very aligned on our cause-driven work, that was part of our initial connection.”
Harry and Rachel believe that life is unpredictable and a fairy tale may not be the perfect way to understand the royalty. “There was no way to understand what the day-to-day was going to be like,” says Rachel.
Since their very public marriage, Harry and Rachel have received a unique but often unbalanced combination of privilege and scrutiny. In response to these pressures, the couple have embraced their authenticity and vulnerability, using that to support the work they do.
“It’s so different because I didn’t romanticise any element of it,” shares Rachel.
Harry and Rachel have set a new standard for Royalty and their approach of balancing a life in the public eye with a commitment to real acting out their beliefs has been a model of courage and grace.
It reads quite nicely (strings-together quite fluently), but…
It’s quite fawning. It’s again kind of endorsing/reflecting what the interviewees are saying. It’s like something you’d read in Hello magazine. There’s no journalistic distance from the subject. Whether a journalist with 20 years experience (like me) would do that depends entirely on the nature of their work and employer.
That’s particularly noticeable in the final paragraph, which incidentally turned out to be the opposite of how some people think Harry and Megan have turned out to be. It’s like a PR piece - but then, that was the setup with the Oprah interview.
I ran the same prompt on one of my transcripts, which is business-flavoured.
The result is more boring and plain. I’m sort of happy that the AI didn’t go out on a limb and make assertions in its own voice on this occasion, but, again, what is left is a quite prosaic series of he-says attributions. There’s no invention, no turn of phrase.
Well, I think that’s where finetuning really would help. If you have a couple of examples of articles you’ve written and interview transcript, you could try to make the AI understand implicitly what you want from it. Have you tried fine-tuning before.
Well, I think that’s where finetuning really would help. If you have a couple of examples of articles you’ve written and interview transcript, you could try to make the AI understand implicitly what you want from it. Have you tried fine-tuning before.
I agree… I would like to think fine-tuning is the best and ultimate way to do replicate what I do.
I do have an archive of thousands of my articles similar to this, which I think would constitute an excellent training set.
I have an ambition to train/fine-tune the AI on them.
But a) I’ll need to learn some fine-tuning craft first and b) I’d like to make my archive available in a more future-proof, plaintext format that could also end up facilitating this project.
I think I need to scope-down that ambition and just try out a smaller set.
I’ll be interested to see how the fine-tuning uses it all. ie Does it it take the structure/style/approach of my sample articles as the thing to emulate, or does it take the content and subject matter? The former is what I’d want, as the content would naturally be different every time (although in the same broad industry).
It’s been a while since I checked in on this stuff, and OpenAI has now launched the “Chat” mode in the Playground/API.
Anyone think “Chat” mode offers some pathway to improving this?
Would I stuff into System all the guidance about the writer-bot, how it should behave etc? And then just leave the messaging area to a basic “write an article from this transcript” prompt?