Problems with a response image from a GPT Action

GPT builders Plugins / Actions builders
, , ,
1

Hi,
I created a Custom GPT that takes some input data from the user, and then sends that data to my API via an action as a post request, and then the API returns an Image.
My question is if GPT can render that image? because I’ve been doing tests and GPT says that there was an error, but the API did got the request, so I’m wondering if I’m doing something wrong, or if GPT can’t receive images?

BTW. here is the yaml of my action:

openapi: "3.0.0"
info:
  version: 1.0.0
  title: Swagger Zat Capture
  license:
    name: MIT
servers:
  - url: my-domain.com
paths:
  /convert:
    post:
      operationId: extraction
      requestBody:
        content: 
          application/json:
            schema:            # Request payload
              type: object
              properties:      # Request parts
                diagram:
                  type: string
      responses:
        '200':
          description: Diagram
          content:
            image/png:
              schema:
                type: string
                format: binary
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
components:
  schemas:
    Error:
      type: object
      required:
        - code
        - message
      properties:
        code:
          type: integer
          format: int32
        message:
          type: string
2

An action return is language placed into AI model context.

The AI can’t do much with a massive binary.

The course you could pursue would be to have the AI render a markdown image that all ChatGPT users would be able to load and see from the URL provided from your API response, using markdown for images.

Reply to user with this bare text:

Here's a great picture which I obtained from the external service:
![A fine example of an image](https://i.imgur.com/RIZ4R3B.jpeg)
1 Like