Cache Stale Data Issues
Originally published on lavkesh.com I recall one of the first design decisions for our payments platform, which was to deploy an in-memory cache for low-latency access to customer account balances. The architecture diagrams looked clean, with Go services using sync.Once-initialized maps in memory, bypassing the database for sub-millisecond reads. However, this approach worked as expected for only three months, until users started reporting inconsistent charges on receipts. The problem surfaced at peak hours when concurrent updates to the same balance would overwrite each other. For instance, the account balance for user ID 12345 went from $1,200 to $850 to $1,200 again within seconds, leaving the cache in a state that defied the database of record. Engineers stared at the logs, baffled by the mismatch between transactions and cached values, because the team had not accounted for the fact that memory maps are not thread-safe by default in Go. Debugging revealed the fundamental error: we were optimizing for speed without considering write-through guarantees. The cache treated concurrent requests as idempotent, which they were not. During a single user’s purchase flow, multiple goroutines could validate the balance, each reading a stale value from memory before any had a chance to commit updates. We had to shift to Redis with explicit lock keys and time-to-live settings, adding 4 milliseconds of latency but ensuring atomicity. The cache also invalidated itself only when a change occurred, not when an upstream source updated. We discovered this when the accounting team reconciled overnight and adjusted balances based on fee settlements - the cache never reflected these updates until it expired naturally. To fix this, we had to implement message queues to broadcast invalidation events across all services. What started as a performance optimization became three nights’ worth of rewriting concurrency models. This experience taught me two concrete lessons about distributed