Hello, I am working on a project that requires a large pdf to be processed and exact data to be retained. I want to take a course catalog (1500 pages) and turn in a chatbot that you can ask questions related to courses. Currently my problem is that with a large amount of pages the answers tend to b…

You should properly parse tabular data. There is definitely packages for that. After that you can go multiple ways: You can just put in the tabular data in your vector db (make sure the table data is contained within one chunk with a header). You can take each value of the table and generate an ex…

Are there also learning paths included or just courses? If only courses you should split the PDFs in single pages, then create some overlap (1/3 of the page maybe) to find the end and beginning of a course in the first step. Then embedd the courses one after the other. Although I think there are …

I turn pdf into text and then chunk it up page by page. Found some great video while browsing other post in this forum. If you search “ GPT-4 Tutorial: How to Chat With Multiple PDF Files (~1000 pages of Tesla’s 10-K Annual Reports)” on YouTube you should be able to find it (sorry, cannot post links…

This is whole lot of new information to try out. Thank you! There is in fact learning paths (I call them sample plan of study). These plans are tables in the pdf, but when the document is turned into text it just all goes sideways (doesn’t help that the original pdf has 2 columns). Why would you re…

Use ghostscript to make tiff from pdf, then tesseract to create hocr from tiff. Read about hocr.

Looks very promising, thank you. And your initial message implied that you would do something else differently with sample paths included?

Working in that for a couple months. So yay, differently. A little :sweat_smile:

Hey, did everything you described. I now have 1500 separate .hocr files. What do I do with them? Convert to text?

Run them through GPT to identify the end of a course. Hocr is a good format for gpt to understand. You may try hocr2yaml as well to save token.

Sorry to ask too many questions. But I believe this method will not work for me. The problem is that there are two columns, and alphabetical output turns into a mess. Here is the example: Ancient Near Eastern and Egyptian Studies, BA… 284 Mathematics, PAD eee SIS Ancient Near Eastern and Egyptia…

Using large PDFs to make a ChatBot

API

ArtZab September 14, 2023, 4:37pm 13

Found your post here API for Advanced Data Analytics - #2 by jochenschultz . Will try it out.

Topic		Replies	Views
Retrieval Augmented Generation (RAG) with 100k PDFs?! Too slow! Community pdf , llm , rag , development	13	30100	October 31, 2024
Creating a bot using 100+ PDFS as the knowledge base API	18	18055	June 25, 2024
Problem with doing RAG with 300k pages of PDFs Community gpt-4 , gpt-35-turbo , api	8	6573	March 7, 2024
Train GPT for analyze large number of pdf Community chatgpt	8	2853	August 2, 2024
Efficiently Interacting with super super Long PDFs/documents API gpt-4	2	1645	June 25, 2024

Using large PDFs to make a ChatBot

Related topics