CHAT-GPT Search API For Document Upload

API
1

Hello,

I have a website with over 2,000 documents that I want to make searchable. I’ve tried a couple of techniques to achieve this, but I ran into some issues.

The first approach I tried was to convert the documents into text files and send along with the search query, to an completion API . However,
I found that this method failed when I had documents with more than 3,000 to 4,000 words, and I received a maximum token exception.

The second approach I tried was to convert one document into a JSONL file and upload it using a file endpoint. This method worked well,
but I don’t want to use it because it involves fine-tuning a model, and I have a large amount of data that I want to search through.

Is there any procedure to achieve that?
I am using .Net Language but i can convert it if any body knows it will be helpful.

1 Like
2

Hi @dev.umaralifarooq

You can use embeddings.

There’s a tutorial as well for a similar purpose.

1 Like
3

We also now have a specific tutorial on doing this with files: openai-cookbook/apps/file-q-and-a at main · openai/openai-cookbook · GitHub

4 Likes
4

@logankilpatrick thanks for your help, i don’t have access to download the next.js code?

5

@sps Thanks for your reply, do you have any other help because its much confusing.

1 Like
6

Check my post, I think it can help you! I explained in detail how embeddings work.

7

Hey, did you figure it out? I am struggling lol

8

Only to search, you can use solr. It can process fast all kind of documents (pdf, doc, txt, html, etc.)
You can use chatGPT API to convert spoken language search queries to solr queries. ChatGPT can act as an interpreter ( human->solr )