Hi all, Hope you can help me with this. I have a functional Assistant API built with Node.js, and I’m looking to enhance the user experience by enabling users to add images within the same conversation. The assistant will analyze these images and seamlessly continue the conversation based on the an…

My PR (albeit in RoR) might help: [image] Does the model `gpt-4-vision-preview` have function calling? API So I gave up and wrote a function. With hindsight this has one huge advantage: you can use a cheaper LLM to hold the general conversation and restrict calls to GPT …

Thanks @merefield Can you maybe explain the logic steps? Send image to the backend Have the Vision function seperate When the backend route receives an image from the frontend, trigger the Vision function and analyze the data Stream the data back to the frontend Is this also your approach?

I’d appreciate if you could please read the code. Should be very readable and clear.

I have a little more time now. Apologies if I was a bit blunt, but I have would appreciated if your question would have demonstrated you had read my code. I wasn’t convinced! This would have saved me time if you could have been more specific in your question. No, that’s not the flow with my (Dis…

Hi @merefield , Appreciate the message! I was just confused by your Github Repo, so was curious to hear your approach. Anyway, thanks for the reply :slight_smile:

Yeah the whole repo would be overwhelming, that’s why I shared the PR. If you look at “Files Changed” on that PR it may be easier to digest.

If anyone is interested in this, I got it working Let me know if you want to see the code snippet! Next up: text-to-speech :joy: Offtopic: do you guys think Vision and text-to-speech will be integrated into the assistant-api’s any time soon?

Hi, can you share the code. I am creating a chatbot where I want user to upload an image during a conversation, I have different processing pipeline for image and I do not want GPT4V. I am confused about how should I validate whether the user has uploaded an image or not.

Yes sure, here it is: router.post(“/assistant-chat”, async (req, res) => { const { threadId, imageUrl, contextText } = req.body; let { userMessage } = req.body; try { let currentThreadId = threadId; // If a threadId is provided and exists, continue, otherwise, create a new thread if (!current…

And here is the vision api: async function analyzeImage(base64Image, contextText) { // let promptText = “analyze this image as if you are an experienced interviewer at a top management consulting firm (MBB). Provide detailed feedback on how you would evaluate this in the context of a case intervie…

Integrating Vision with Assistant API

API

merefield March 27, 2024, 6:54am 2

My PR (albeit in RoR) might help:

Topic		Replies	Views
Can Assistants API understand image files uploaded? API	11	11961	September 28, 2024
Which is correct model for image analysis? API	5	442	December 9, 2024
Inputting an image in the Assistant API using the new vision model API gpt-4-vision , assistants-api	9	5428	July 16, 2024
Unable to directly analyze or view the content of files like (local) images API chat-completion , gpt-4-vision	3	1288	November 7, 2024
Asisstant API for querying image API	5	559	June 25, 2024

Integrating Vision with Assistant API

Related topics