We’ve got an AI chatbot based upon OpenAI’s models, and we’ve got context we’re extracting using VSS and RAG. I want the AI chatbot to display relevant images as Markdown, but almost regardless of what I do, OpenAI returns only text, and it’s very difficult to ensure it’s returning , even though I explicitly send it such information.
Advice …?
Yeah, if you query the text API, you’ll get text back. The image API is separate.
Are you wanting the API to generate image + text? If so, you’ll need to make multiple calls likely.
What are you trying to do exactly?
How are you sending the images you want to be displayed back to the API?
Not sure, how you are implementing your RAG. Let us assume you are using function calling, so here is a possible way:
Here’s the raw output in markdown:
I’m sending Markdown to OpenAI. This Markdown contains image references. I want to make sure it returns this Markdown consistently. I don’t want OpenAI to generate images, just return existing images as Markdown.
Sure, we’ve got an AI platform. Think of it like “an administration tool for OpenAI and LLMs”. One of its features is we do RAG and VSS search to find “context” we send to OpenAI for our chatbot - You can try it below if you wish. One of our unique features is that we can display images inline in the chatbot itself, which is a big deal for technical support solutions (screenshots, etc) and e-commerce (product images, etc)
My problem is that OpenAI is not consistently able to preserve the markdown we’re sending it, which contains images and hyperlinks. It’s really more of a “prompt engineering thing” than a technical issue, I’m hoping somebody else here have already solved, as in how to create a system message / instruction that allows me to transmit Markdown to OpenAI (using their API), and have OpenAI return relevant images / hyperlinks it finds in the context.
Am I making sense?
You can try our own chatbot below …
