Could be hallucination, but I gave this to an LLM and this is what it suggested:
"
The workflow shown in the video—processing an image and then controlling a computer interface to draw it—is a combination of two well-established fields: Computer Vision and UI Automation.
You do not necessarily need a Large Language Model to perform the underlying image processing; standard algorithms can do this deterministically.
Step A: Image Processing (The "Brain")
You can write a script (using Python libraries like OpenCV or Pillow) to process the reference photo:
- Edge Detection: Use filters (like Canny or Sobel) to find the "high spatial frequencies" (outlines).
- Color Quantization: Use clustering algorithms (like K-Means) in the HSL space to group millions of pixels into a small palette of distinct colors.
- Vectorization: Convert these processed shapes into a set of coordinates (SVG paths) that represent exactly where the mouse needs to move.
Step B: UI Automation (The "Hand")
Once the image is converted into a set of instructions (coordinates and color codes), you can use automation tools to physically control the computer and draw on Canva.
- Browser Automation: Developers have already created projects that use Selenium (a web automation tool) combined with edge detection algorithms to draw images onto HTML canvases.
- The script reads the pixel data, calculates the mouse coordinates, and executes the "click-and-drag" actions in the browser.
- Computer Use APIs: In the case of GPT-6 Astra, the model uses a "Computer Use" interface. It effectively takes the processed image data (or generates it internally) and outputs high-level commands (e.g., "Move mouse to X,Y," "Click," "Select Hex Color #FF5733"), which the system then executes on the screen.
"
Seems plausible and easier to believe. Also, feels like a "magic trick" designed to fool the user into believing that the agent is drawing interactively by using its vision, since it could just have written a python script that takes the input image, and produces the exact same result without automating the screen.
