Responses.create rewrites image URLs and returns expired SAS blobs in my MCP tool call

API
, ,
1

Hi everyone

I’m building a bot that chats with users through the Assistants API.

When a user attaches a photo, the bot should forward that image to our internal MCP micro-service for analysis (OCR, object detection, etc.). I’m orchestrating this flow with openai.responses.create and a tool definition for mcp.process_image.

Current architecture

  1. Client → S3 – the browser uploads the image and receives a presigned GET URL.
  2. Gateway → OpenAI – the gateway calls
openai.responses.create({
  model: "gpt-4o-mini",
  tools: [ mcp.process_image, … ],
  messages: [
    { role: "user",  content: "User prompt…" },
    { type: "image_url", image_url: { url: "<my-presigned-S3-URL>" } }
  ]
});
  1. Assistant → MCP – if the model decides to run the tool, the platform forwards the call to mcp.process_image.

Problems

  • Own storage path: inside the tool call I never receive my original S3 URL; responses.create rewrites it to something like oaiusercontent dot com…, so the MCP can’t fetch from our bucket.
  • Base-64 fallback: if I embed the image as data:image/png;base64,…, the MCP still gets an OpenAI blob URL — but the SAS token inside that link expired on 2023-12-06 13:31 UTC, so Azure replies AuthenticationFailed.
  • files.create experiment: uploading first with openai.files.create doesn’t help; the ID that reaches the MCP sometimes differs, and when it matches, GET /v1/files/{id}/content returns 404.

Question

How can I either

  1. force the Assistant to deliver my original presigned S3 URL to the tool untouched, or
  2. get a blob URL whose SAS token is still valid when the MCP receives it, or
  3. reliably use files.create so the tool can download the image?

Any best-practice examples or code snippets would be hugely appreciated. Thanks!

image
image1448×548 75.8 KB

1 Like
2

The field value for image_url/url is never seen by the AI language model. It is used just by the API to retrieve internet data.

(it also would be bad news to put a base64 data URI there to “upload” an image, and then have an AI see the string as language).

The first part of having the AI accurately recite a URL into a tool field is to provide that URL in the prompting.

You also show construction of the “Responses” endpoint API call wrong, in an imaginary metacode manner, as the field for sending a list of messages is called “input”, each has a role name, the only role that can include images is “user”. The user message is an array of multimodal parts by type.

Actual message construction then, for understanding and further use would be best constructed and communicated like:

system:

  • text: You are an AI assistant with internet-based image analysis tools beyond your own ability.

user:

  • text: Locate Waldo, giving the center pixel coordinates.
  • text: user attached image 1 from ~https://waldogames.ai/3355.png~
  • image_url: (image data)
  • text: (end of images)

If you can’t give unauthenticated blob storage for images or temporarily re-host, you can provide images as input to generation requests in multiple ways:

  • By providing a fully qualified URL to an image file
  • By providing an image as a Base64-encoded data URL
  • By providing a file ID (created with the Files API)
3

Adding the pre-signed url to a role: user message solved it! Thanks