Hi everyone, I’ve been trying to automate the inputting of prompts that adhere to the 4096 token limit. I’m doing this mainly for text summarization. What I’d like to ideally do is take a giant body of text and split them into chunks of 4096 tokens, send them into a chatgpt with a very generic promp…

Sign up for the API and you can use Python to submit as many requests (as you want to pay for)

I believe @daveshapautomator created a Python script to do this.

My notes on the subject: Summarize Large Documents How to Summarize a Large Text with GPT-3 How to Summarize a PDF file with ChatGPT (70 000+ Words) State of the Art GPT-3 Summarizer For Any Size Document or Format | Width.ai Smaller chunks allow for more understanding per chunk but increase th…

Welcome @jzhanglsw gpt-3.5-turbo has a context length of 4096 tokens meaning that is the max tokens it can process(read + generate). If you send it 4096 tokens(original text + prompt), there’s none left for generation. Summarization of text bodies that have near or greater than context-length toke…

I’d imagine this would get expensive if I wanted to summarize whole books. I have chatgpt plus so the normal chat I can use as much as possible. I just need to automate putting them into the actual text box.

At the time of input, I actually don’t ask it to summarize anything. I just tell chatgpt that I’m giving it a large body of text and to say okay. I did an experiment where I split the first chapter of Harry Potter into 2 chunks that reach near the 4096 token limit (the first chapter is about 6471 to…

Any chance you can point me to the repo? Searching chatgpt or token on his github result in amusing but irrelevant code repositories.

Could be because of the model’s preexisting knowledge of the book.

You have a valid point so I tried the experiment again with the following article written in April 2023: (Apparently I can’t send links so just search the putin myth journal of democracy on google) Again, the summary was quite good. I believe that despite the token limit, somehow chatgpt is able t…

Ways to automate breaking a large piece of input into chunks that fit in the 4096 token constraint?

API

aikanarov May 21, 2023, 4:02am 3

You can use many techniques like map reduce, refine, etc… I use langchain for it.

Summarizing and extracting structured data from long text

Topic		Replies	Views
Logic behind uploading a large document API chatgpt	8	1849	June 4, 2024
Prompting with the chat/completions API against a large transcript file API	5	3817	October 4, 2023
The length of the embedding contents API	47	36176	November 2, 2023
Summarizing and extracting structured data from long text Prompting gpt-4 , api , token , limitations	14	13837	February 19, 2024
Preprocessing - I just don’t get it! API	14	3356	January 6, 2023

Ways to automate breaking a large piece of input into chunks that fit in the 4096 token constraint?

Related topics