How to send long articles for summarization?

API
11

Hello and thank you for your responses.

In the end, I used the idea from this image: https://www.allabtai.com/wp-content/uploads/2022/12/big-file-summerize-gpt3.jpg

The difference is that the image suggests sending the paragraphs and then combining all the summaries at the end.

Here’s what I did:

  1. Combine the paragraphs based on the number of tokens.
  2. Send them to create a summary.
  3. Combine the previous summary with the following paragraphs based on the number of tokens.
  4. Send them to create a summary.
  5. Go back to step 3.

This way is better because in each generation, I send the context rather than just isolated blocks of paragraphs that may not make sense with the complete text at some point.

1 Like