标签：#uptime

If you just want the recommendation: build the uptime poller yourself, put exponential backoff with jitter in front of every health check, and treat a 429 as a scheduling signal instead of an error you swallow. Query-style observability APIs hand you metrics and logs, not threshold rules or notification channels, so the polling worker is the thing that has to decide what "down" means and who gets woken up. That decision is the whole job. I got burned by exactly this. What follows is the pattern that survived the postmortem, the alternatives I weighed before writing a line of it, and the conditions where you should not do any of this yourself. The 429 my retry loop ate for six hours Last spring I was running a homegrown health checker for 40 internal services. One goroutine per service, all driven off the same 15-second ticker, which meant every check landed inside the same 200ms window. The status API we polled had a per-minute quota I'd never bothered to read, and for months it didn't matter, because 40 checks a minute sat comfortably under the ceiling. Then a colleague onboarded 12 more services, we crossed the quota, and the API started answering with HTTP 429. My retry wrapper caught it, retried three times in a tight loop, and on the last attempt returned the previous cached result — which said healthy . It logged the rate limit at debug level. Nobody reads debug. Six hours. Green dashboard. Dead queue consumer. We found out when a customer asked where their export was. The consumer had died on an unrelated deploy, the checker never noticed, and when I finally restarted it the backlog got re-processed on top of a manual replay I'd already run — two customers got the same notification twice. Duplicate deliveries are the specific thing I lose sleep over, and I had caused a batch of them with a retry loop that was trying to be helpful. The postmortem produced one line I now paste into every runbook: a check that can't reach the API reports unknown, never healthy.