OpenAI Embeddings - Search through ~1000 PDFs

API
1

Hey everyone, I’m new to AI world and I’m a bit unsure if I’m in the right spot, but here goes.

I’ve got this puzzle I’m working on with a pile of small PDFs—like 1000 of them. They’re short, around 5-7 pages each.

What I want to do is use some embedding magic to turn these documents into vectors and stick them in a database. The plan is, when I toss a question into the mix, I can scan this vector database to find the closest match to my query. his extra context should help OpenAI serve up a spot-on answer.

I have to questions regarding this:

  1. I want the response to include the document name or title where the answer was snatched from. (Is this even possible?)

  2. I’m curious if I can cut down on the embedding expenses by creating these vectors only once and saving them in a database. That way, when I fire off a new question, I can simply embed the prompt and hunt for the closest match in my pre-existing vector stash. And then use this result as a context to my question to llm?

3 Likes
2

Hi! I need to develop something similar. I was wondering how did you en up doing it. Thanks

3

Update: I use a number of text extractors now: PyMuPdf, AWS Textract, PdftoText, Solr tika, Marker… I even use models sometimes to extract text from images and/or pdfs: Gemini 1.5 Pro and GPt-4o.

My semantic chunking process has also evolved significantly: Using gpt-4 API to Semantically Chunk Documents - #166 by SomebodySysop

If all of this is too much, here is a tutorial I think covers the basic nuts and bolts of the embedding process: https://www.youtube.com/watch?v=Ix9WIZpArm0

1 Like
4

You can use this tool with OpenAI embedding model

1 Like