Recursive Summary Algorithm for Large Files

API
1

I am pondering if it is possible to summarize a large document by breaking it into pieces small enough to submit to GPT.

Then concatenate those summaries and repeat the process on the new smaller document.

Then repeat until satisfied with the output.

Has anyone done this? Is there either a commercial or an open-source project which already does it so I am not re-inventing the wheel if I do it myself?

2

Embeddings can be used in this situation.

You can check out GitHub - mayooear/gpt4-pdf-chatbot-langchain: GPT4 & LangChain Chatbot for large PDF docs for more information on the open source project.

Perhaps As ChatGPT only summarises the most recently typed 3,000 words, I wrote a code to do it for every 3,000 words | by The Tech Insider | Geek Culture | Apr, 2023 | Medium is better suited for your needs.

1 Like
3

Perfect - thank you. Those are Good resources