PDF summarizer using openai

Embeddings are usually used so that we can retrieve chunks of text for an retrieval augmented generation (RAG) application. For example, given user query A, I want to find documents related to it. This process of “finding documents related to it” is done by comparing the embeddings of the Query A & your repo of documents

From what you’ve described, your scenario is much simpler: You’re just summarizing a given text in a pdf. If the length of your pdf exceeds the context window of the model, you can chunk it up into smaller parts & ask the LLM to summarize each smaller part.

As a first step, increasing the max_tokens parameter as others have suggested & also checking the token length of your document. Consider looking at this other post: Counting tokens for chat API calls (gpt-3.5-turbo)

1 Like