Using gpt-4-vision-preview in Langchain

Community
,
1

I am trying to create example (Python) where it will use conversation chatbot using say ConversationBufferWindowMemory from langchain libraries. User will enter a prompt to look for some images and then I need to add some hook in chat bot flow to allow text to image search and return the images from local instance (vector DB)

I have two questions on this:

  1. Since its related with images I am trying to use gpt-4-vision-preview model in my code. My code samples looks like:
from langchain.chat_models import ChatOpenAI

qa = ConversationalRetrievalChain.from_llm(ChatOpenAI(model="gpt-4-vision-preview", max_tokens=1024), retriever, memory=memory,chain_type="stuff")

It gives me the error: The model gpt-4-vision-preview does not exist or you do not have access to it. Learn more: How can I access GPT-4? | OpenAI Help Center.

However in my other code sample if I do it like:

import openai
 response = openai.chat.completions.create(
        model="gpt-4-vision-preview",
        messages=[
            {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the image in detail"},
                {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/jpeg;base64,{image_base64}",
                },
                },
            ],
            }
        ],
        max_tokens=4096,  # default max tokens is low so set higher
    )

In this case it works using my openAI key.

So does gpt-4-vision-preview model is supported only in method openai.chat.completions.create() ?

  1. Other question is what I am trying to achieve is really feasible / doable? I mean when I go though the open AI documents I got many samples where is user is trying to read and attach PDF or other docs to a langchain ConversationalRetrievalChain and do question answer session.

But when it comes to say text to image search using Weaviate existing Schema (with images vectorized) can I allow the text to image search in conversational chatbot?