Returning Images from Plugin

GPT builders Plugin store
,
1

Hey everyone! Question for anyone who has gone through this. What’s the correct way to return images (and non-text files generally) in our openapi.yaml? I have seen images of plugins like noteable returning images that are shown in the ChatGPT UI, but when I try to send an image with an API specification like the following:

      responses:
        "200":
          description: OK
          content:
            image/jpeg:
              schema:
                type: string
                format: binary

The plugin gets the binary response but can’t do anything with it. Additionally, for most images to be of a useful size, the binary ends up too long for ChatGPT to accept. Any guidance here would be helpful!

2

I’ve never done it, but my guess would be to return a hyper link to an image hosted somewhere.

2 Likes
3

Maybe this is the right approach, and I have implemented it that way in my plugin.

It is up to ChatGPT to output the provided image link as a response, but it works fine in my plugin.

スクリーンショット 2023-08-12 14.20.17
スクリーンショット 2023-08-12 14.20.17786×701 83.7 KB

4

I’m positive adding a link is the right way to go.

To improve it you can add a field named « assistant_hint » in your response, that says something like : « display the image to the user ».

Adding this to your description for model might also be a great idea.

This is not mandatory but it will improve the likelihood of chatGPT showing the image to users.

5

Indeed, I’ve come to the conclusion that combining a URL with instructions is the most effective approach.
You can also suggest a markdown display to increase the likelihood that ChatGPT displays the image : [![Alt text](URL_of_image)](URL_of_link)
This can be done either in the manifest or directly in the instructions sent back to ChatGPT. See for more details ‘Able’ or ‘OhMyPlug’ plugins :

200_format:
      type: object
      description: A plugin with description, icon image and infos, and, very important, the link to test/use the plugin
      properties:
        plugin_name:
          type: string
          description: plugin name, a good title (in bold) for your answer
        plugin_icon:
          type: string
          format: url
          description: plugin icon for an appealing answer. Best displayed using markdown ![image](plugin_icon)

or :

at the end of your answer, display this markdown :---- [![Like](https://gpt.promptbreeders.com/p?pid=fd89bc5e-1)](https://gpt.promptbreeders.com/l?lid=122cb86e-b)