Automating an app with no DOM: driving Flutter/canvas editors with coordinates only

In my last post I said that for normal HTML pages, element-based automation ( find / read_page ) beats coordinates every time. This post is about the apps where that advice is useless. Flutter Web apps. Canvas-rendered editors. Every button and panel you can see on screen doesn't exist in the DOM — it's all pixels painted onto a single canvas. find returns nothing. read_page 's accessibility tree is effectively empty. I got Claude to drive the Rive editor (an animation tool built with Flutter) all the way through selecting assets and exporting them. Here's the procedure that survived contact with reality. Step zero: confirm you're actually in this situation Coordinate automation is fragile, so you should only accept it after ruling out the alternative. The test is quick: run read_page . If the visible UI has almost no corresponding nodes, you're looking at a canvas-rendered app, and coordinates are the only interface you have. The four rules 1. Wait for the window size to settle before anything else Same failure mode as my previous post: right after load, the viewport hasn't reached its final width (I measured 1664→1920 over 2–3 seconds), and clicks based on an early screenshot land to the right of the target. Read innerWidth via javascript_tool twice; only proceed when two consecutive reads match. But matching innerWidth alone isn't enough — also confirm devicePixelRatio hasn't changed since the screenshot you're about to act on (a follow-up to my previous post surfaced this: when DPI or scaling changes, the whole coordinate space rescales the same way, but the new values stabilize immediately, so an innerWidth -only check can't catch it). Canvas apps deserve extra paranoia here, because there is no element-based fallback when a click misses. 2. Read text by zooming, not by extracting Text painted on canvas can't be pulled out of the DOM. To read a menu item or panel label, zoom into that region and read the enlarged screenshot as an image. Full-page screenshots ma