Summarizing and extracting structured data from long text

Prompting
, , ,
16

I have searched for a long while and got 2 ways to deal with this situations.

  1. Splitting the document into multiple chunks and embed in MapReduce or Redefine way (which could not stuff the whole document into a message content window size) Summarization | 🦜️🔗 Langchain

  2. Using the tools such as ChatGPT Prompt Splitter to split your document into multiple parts and send them by multiple messages. follow the blog article. Ways to automate breaking a large piece of input into chunks that fit in the 4096 token constraint? - API - OpenAI Developer Forum

While I have no idea which is the better. according to my testing. I thought the Map-Reduce way would generate a better result. As I know the method 2 seems like the Refine way. Their workflows are similar