Debugging is also clicking 🖱️

In the last couple of posts I let agents debug over DAP — breakpoints, step over, continue. That's real debugging. But it's only half of it. When I debug something for real, I also click : I press the button and watch what happens, read the dialog, notice the toggle is greyed out. No backtrace ever tells you the Save button never enabled. So — can the agent do that half too? The web is the easy case Browsers are automatable by design. Most agent tools ship their own browser or drive an external one; point Playwright at a page and every element has a stable, queryable handle. The DOM is an accessibility tree wearing a different hat — roles, labels, structure, all there for the reading. For the web, this half of debugging is close to solved. Native apps are another game There's no DOM. When the agent has nothing to go on, it falls back to the eyeball approach: take a screenshot, let the model look, maybe run OCR or a pre-analysis pass to label what's on screen. It works — and sometimes it's the only option — but it's brittle (a few pixels off and the click misses) and it burns tokens describing pictures. I ran into this by accident. I once wrote a tiny skill whose only job was to screenshot a running 4D form and stitch an animated GIF for a README — 4d-capture-gif . Then I noticed Claude Code reaching for it to debug : the skill also reports a bit of the form's structure — where the buttons are — so the agent knows where to click. For simple cases it genuinely works. But screenshots-plus-coordinates is not the thing I want to build on. The cleaner path: read the tree, don't look at pixels Instead of staring at the screen, read the UI tree directly. On macOS you can script the Accessibility API from Python (pyobjc), and there are automation libraries to help. Now you're clicking element #37, the "Save" button instead of coordinate (412, 260), and hope . A couple of open-source tools are pushing exactly here: agent-desktop — a native CLI that exposes any app's accessibi