Paraphrasing with GPT-3

I’ve been trying to get GPT-3 to paraphrase passages of text, with no success.

With all prompts i tried, at low temperatures, GPT-3 repeats the input text. At high temperatures, it starts to produce factually incorrect sentences. For the use case I have in mind, the model needs to produce factually correct output in at least 95% of the time. But the text should also be sufficiently different from the original.

Example prompt
Paraphrase the original passage.
""""""
Original passage:

""""""
Paraphrased passage:

Has anyone cracked paraphrasing? I also tried one-shot and few-shot prompts but with similar results.

2 Likes

I’ve had great success with CURIE-INSTRUCT. The prompt is simply “Generate a very detailed summary of the following:” and then a few examples.

2 Likes

That’s an important note to keep in mind, @m-a.schenk!

@daveshapautomator that’s a great prompt. I’ve had a lot of success using davinci-instruct-beta with no examples at all (with penalties to the max to avoid repeating the prompt, and a medium temperature).

5 Likes

did you use stop words, start words? I mean, what would be the details of the parameters Joey?

Just the default parameters, besides what I described.

1 Like

Thanks joey! My prompts returns similar results.

The issue is, that as soon as the model “coincidentally” returns one sentence, that is the same as in the input, all the remaining sentences are exact copies (which seems natural given the “predict next word” objective of the model). See e.g. the last two sentences in your output, they match the input exactly.

Glad to help!

This is just a starting point, and I’d definitely add high-quality, varied examples to help the API.

Edit: To add to that, you could also re-phrase one sentence at a time, circumventing the problem of entire paragraphs not being properly re-phrased. That said, this is entirely different from the earlier suggestion of providing examples, so I’d try both out, depending on your exact needs.

1 Like

Informal: Hulu has a big user base.

Original: Hulu commands a sizeable market.

Rewrite: (Keywords: wields, reach) Hulu wields a vast reach.

Informal: Milk is good for your bones.

Original: Milk strengthens one’s bones.

Rewrite: (Keywords: documented, bolster) Milk has been shown to bolster one’s bone strength.

Informal: GMOs are seen as bad, but that’s not accurate.

Original: GMOs command a negative reputation, but this sentiment is inaccurate.

Rewrite: (Keywords: tainted, inaccurate) GMOs bear a tainted legacy, but this assessment is unfounded.

3 Likes

This is a pretty good prompt! :+1:

If you’re paraphrasing a list of items, GPT-3 won’t shorten the source material. A list can be anything that has similar meaning.

For example: “I went to the foyer. Then I put on my shoes. Then I opened the door. Then I…”

Summarization works best when each sentence adds to the overall theme, without adding too much new repeatable information. Think narrative arc.

For example: “He had a tattoo on his left wrist. It was a star from his neighborhood childhood, he told me. He seemed downcast when he looked down at it, as if he was appraising a faded picture of long dead relatives in some attic. I’ve always remembered that tattoo…”

3 Likes

Is there any reason why the Content rephraser is capped at only 30 tokens?

A lot of times, our input sentence easily exceed that - and it’s precisely why we need the AI’s help to rephrase and break down the sentence.

It’s quite unusable at 30 tokens.

1 Like