I want to run queries on custom data that are science papers, around 1000 of them. They have images as well. What is the best approach here?

I would start by extracting the text from the papers and exposing it for search. It could be searched via vector correlation with embeddings, or based on keyword search, or a hybrid of embeddings and keywords. Images are tough right now, but maybe with the Vision API around the corner, even the im…

Thanks, thats what i thought. Plenty of documentation on text. How does it work in training. Or when you ask it to give you an image of something? Is it an external system? Meta data on images?

Maybe best to get llm to describe the image then store that text for image search?

Yes precisely. That’s basically what the Vision component of the API would do. You give it an image, and ask for the text description. FYI, text is the “lowest common denominator” for search. If you have an audio file, you transcribe it to text as well to expose it for search. So you will have …

So whats the status of vision api? Any time lines?

It was just announced for ChatGPT. So my guess is later this year in the API at the earliest, or Q1/Q2 next year. If this is a pressing project, you could see if there are other API’s or open source models suitable for the task of describing images and graphs of data. But no matter what, you pipe…

Maybe google one might help. Not allowed to post link. Bit risky to put a lot of work and if api is coming

Use own data with images for queries

API

kabischmid October 4, 2023, 12:36am 9

Wonder if any of these actually hndles images
ChatPDF.com - Chat with any PDF using the new ChatGPT API, not just ocr?

Topic		Replies	Views
DALL-E API to generate json data from image API api	11	5099	October 14, 2023
Use ChatGTP for semantic search or as computervision and keyword search API	8	1122	October 24, 2023
Make OpenAI Vision API Match GPT4 Vision API chatgpt	4	4208	December 6, 2023
Seeking Help with Image Classification Workflow Using ChatGPT Assistant API	1	1924	December 15, 2023
Knowledge Retrieval: support for PDF images Feedback knowledge-files	9	2348	October 28, 2024

Use own data with images for queries

Related topics