I’m learning Blender and usually keep GPT open in a side window for Q&A as I work. I’m imagining a workflow where GPT could see my screen in real time and provide context-aware guidance.
For example: I’ve created a cube and want to scale it along the Y-axis. Instead of typing a full prompt, I could just type “scale along Y,” and GPT, understanding the visual context, could give precise instructions immediately.
Is there any API or approach that allows GPT to interact with my screen this way for live, visual guidance?