Hi everyone,
I’m trying to build an AI agent that can draw, arrange, and manipulate objects on a canvas from natural language prompts.
My current approach is roughly this:
-
The canvas is rendered from a JSON file, which acts as the source of truth.
-
The agent uses tools to read and update that JSON.
-
The frontend then renders the updated JSON onto the canvas.
-
I was exploring something like the open ai Agent SDK / tool-use approach, where the model does not directly control the canvas UI, but instead calls tools that modify the underlying state.
The issue is that it is not working or feeling the way I expected.
One thing I am unsure about is whether the agent actually has enough understanding of the canvas itself. The user sees a visual canvas, but the agent mainly sees tool outputs and the JSON state. So I’m not sure if the “visual understanding” of the canvas is really reaching the agent in a useful way.
I’m also trying to understand how these types of problems are usually handled when the agent is doing something other than writing code. In my case, code is not the main thing dictating the changes. The agent is meant to operate on a visual/structured workspace.
Any help would be appreciated : ))
