Hi, I’m trying to summarise large tokens of input text using completions to pick out key facts common to my input data. I have PDF RPFs being sent to me in a variety of formats and I want to pick out budgets, scope and key dates (submission deadline, project length, project completion date). I’m …

Check out @daveshapautomator github. He has a great recursive summarizer that uses a python module called textwrap for chunking documents.

[image] GitHub - daveshap/RecursiveSummarizer Contribute to daveshap/RecursiveSummarizer development by creating an account on GitHub. Make sure to watch the accompanying video or it might not make much sense. [Recursively summarize text of any lengt…

Amazing - checking it out now! Thank you. Edit - it’s similar to the approach I’ve been taking. I think the main difference here is that summarising a novel you can sacrifice detail. When analysing certain documents however, let’s say an RFP - you don’t want to lose certain details such as ‘Budget’…

For anybody interested in this topic, I’ve also found a python library (not reliant on OpenAI) which may provide additional insights. [image] sumy Module for automatic summarization of text documents and HTML pages. I will review and summarise findings on th…

I was recently using the recursive summarizer for a legal doc. The person asked me to preserve certain data points like stock price and amount, which was getting lost in the summarization. All I did was change the prompt to say, “preserve…”, which worked great. I wonder if your problem might have a …

Yes that’s a good shout. I have seen wildly different results depending on the prompt. I’ll keep trying with this. I’m also looking at NLTK for better string splitting.

but are you sure that chunking it up doesn’t take away from the “essence” of the full story? that’s what i’m concerned about in using the chunking method.

Long text in and of itself doesn’t tend to have a way of encoding past information in a sort of ‘long term memory’. Nouns tend to be anchors to that end. A single session of GPT-3 will only summarise the text (e.g. current paragraph or chunk that is being input). If you wanted to provide a dense …

It would be nice if OpenAI released an example script about how to replicate what they achieved summarizing the books from Project Gutenberg shown in this article Summarizing Books with Human Feedback

Look for daveshap recursive summarizer. It works Great and produces the exact same results as openAI’s experiment

⬛ Splitting / Chunking Large input text for Summarisation (greater than 4096 tokens....)

API

daveshapautomator June 1, 2022, 9:22pm 3

Make sure to watch the accompanying video or it might not make much sense.

Topic		Replies	Views
Summarizing and extracting structured data from long text Prompting gpt-4 , api , token , limitations	14	13776	February 19, 2024
Best way to create responses that exceed token length Prompting	9	5902	November 19, 2021
New 4-turbo model has a unique limit? Or is this a bizarre hallucation? API	18	4737	January 26, 2024
How do I summarise a block of text larger than the token limit? API	12	9392	January 1, 2023
Poor quality response on trained LLM with pdf files Community gpt-4	29	7224	May 1, 2024

⬛ Splitting / Chunking Large input text for Summarisation (greater than 4096 tokens....)

Related topics