Project Log #17: My Agent Misreads Bank Balances. Here's How I'm Fixing It.
Day 17. OCR on banking apps is unreliable. I built a verification layer that double-checks every number. Day 16 was a milestone: multi-app workflows. The agent copied my bank balance and sent it to Mom on WhatsApp. Three apps. One task. But behind that success was an uncomfortable truth: the agent misreads numbers about 20% of the time. For a message to Mom, that's a typo. For a financial transaction, that's a disaster. Today, I built the fix. The Problem Banking apps scored F on my accessibility audit. No UI labels. No content descriptions. The agent has to rely entirely on OCR to read anything on screen. And banking apps have terrible OCR conditions: Small, condensed fonts for account numbers and balances Low contrast (grey text on slightly darker grey backgrounds) Currency symbols (₦, $, £) that OCR often confuses with numbers Commas in large numbers that OCR sometimes reads as decimals The result? A balance of "₦15,000" sometimes gets read as "₦15.000" or "₦15,00" or "₦15000." One missing digit. One wrong decimal. And the entire task is compromised. The Fix: Numeric Verification Layer I built a verification step specifically for financial data. Before any number gets stored in task memory, it goes through three checks. Check 1: Format Validation The extracted text must match a valid currency format. It must contain a currency symbol (₦, $, £, €) followed by digits, optionally with commas and a decimal point. Anything that doesn't match this pattern is rejected immediately. Check 2: Double-Read Confirmation The agent reads the same number twice—two separate screenshots, two separate OCR passes. If both readings match exactly, the number is accepted. If they differ, the agent reads a third time. If two out of three match, that value wins. If all three differ, the task is aborted with an error message. Check 3: Range Validation The extracted number must fall within a reasonable range. A bank balance of "₦0" or "₦999,999,999,999" is probably an OCR error. The agent