How to pass generated images from conversation into ChatGPT app?

ChatGPT Apps SDK
1

Hello, can anyone help explain how to pass generated images or user uploaded images in a chat conversation to a third party chatgpt app?

The new Adobe apps can do this! They pass the local path in tools and somehow turn it into a uri, the param is nativeImage: '/mnt/data/{image}.png’

This is not documented anywhere that i can find?

Some new widget runtime APIs appeared in the docs recently but these don’t follow the same pattern im pretty sure?

window.openai.uploadFile(file)
window.openai.getFileDownloadUrl({ fileId })

Can anyone help clarify this? Is adobe getting allowed special access to images?

2 Likes
4

@OpenAI_Support any guidance on this would be much appreciated! How is Adobe handling this?

5

I’d also like to know this. Following.

6

你观察到的 /mnt/data/… 路径属于模型或工具运行时的内部临时存储，并不是对第三方开发者开放或可依赖的 API。

像 Adobe 这样的集成方可能通过官方合作或私有实现完成图像传递，但这些并不代表公共 SDK 或 Runtime 接口能力。

对于第三方应用，目前唯一受支持、可移植的方式仍然是通过文档中描述的文件上传或可访问的 URL 输入，而不是直接引用运行时文件路径。

建议以官方文档中明确支持的接口为准，避免依赖未公开或内部实现细节

7

SOLVED!

The solution is _meta[“openai/fileParams”]

  1. Add ‘openai/fileParams’: [‘paramName’] to your tool’s _meta

  2. Use z.any() (or equivalent) for the schema type

  3. OpenAI transforms /mnt/data/image.png → { download_url, file_id, mime_type, file_name }

server.registerTool('upload_image', {
  inputSchema: { image: { type: 'any' } },
  _meta: {
    'openai/fileParams': ['image'],  // ← The key!
  },
}, async (args) => {
  // args.image is now { download_url: "https://...", file_id: "sediment://...", ... }
  const response = await fetch(args.image.download_url);
  // Process the image...
});

This is briefly mentioned in the SDK Reference but the transformation behaviour wasn’t clear until testing.

1 Like