Project Log #9: My AI Agent Works on My Phone. But What About Yours?
Day 9. Template matching works. But screen sizes, resolutions, and Android versions might break everything. Eight days ago, the agent was an idea. Now it can read text, handle interruptions, and find icons on a screen. But there's a question I've been avoiding: does it work on any phone other than mine? The Cross-Device Problem Every screenshot I've taken, every icon I've cropped, every coordinate I've mapped—it's all on one device. My phone. Same screen size. Same resolution. Same Android version. Same DPI. Template matching relies on reference images that look exactly like the target on screen. Change the screen density, change the icon size, change the font scaling, and the match confidence drops. Suddenly "send_button.png" doesn't match anymore, and the agent can't press send. This isn't a bug in my code. It's a fundamental challenge in computer vision: reference-based matching breaks when the visual context changes. Today's Experiment I tested the same agent on a friend's phone—different manufacturer, different Android version, slightly larger screen. The results were humbling. Task My Phone Friend's Phone OCR (text recognition) ✅ 95% accuracy ✅ ~90% accuracy Find "Mom" in contacts ✅ Found ✅ Found Template match: send button ✅ 94% confidence ❌ 62% confidence Template match: back button ✅ 91% confidence ❌ 58% confidence OCR held up reasonably well because text is text. Fonts might change slightly, but the characters are the same. But the icons—the send button, the back arrow—were rendered at a different size and slightly different pixel arrangement on my friend's device. The agent failed to send the message. Why This Matters An AI agent that only works on one phone isn't an agent. It's a script. If I want this to be useful to anyone else—or even to myself if I change phones—it needs to be device-agnostic. Possible Solutions I'm Exploring Solution Pros Cons Multi-resolution icon library Simple. Just crop icons at different DPIs. Tedious. How many variants are eno