标签：#monitoring

A status page has a blind spot. It can tell you that your API is returning 502s. It can tell you that a TCP port stopped accepting connections. It can tell you when the incident started. It usually cannot tell you why . Was the application host out of memory? Was disk I/O saturated? Did load climb for 40 minutes before users noticed? Was the server completely healthy and the real problem somewhere else? Those answers often live in a separate monitoring product, disconnected from the incident timeline and disconnected from the status page. That is why I built Servers for StatusPage.me. It is a small, customer-installed host metrics agent and dashboard. You install it on a machine you operate, and it reports CPU, memory, swap, load, disk, and network metrics back to your account. The important part is not “now there are more graphs.” The important part is seeing an outage and the host evidence around it on the same timeline. External checks answer one question. Host metrics answer another. Regular uptime monitoring is still the right tool for the outside-in view: Can users reach the website? Is the API returning the expected response? Does DNS resolve correctly? Is the database port open? Did a scheduled job run? But those checks do not run inside your infrastructure. A healthy HTTP response does not prove that a background worker is about to run out of memory. A timeout does not prove that the app server is overloaded. And an incident can start with a slow disk or growing swap usage long before an endpoint is fully unavailable. The distinction is simple: External monitoring tells you what users can see. Host metrics help explain what the machine was doing when they saw it. You need both. What Servers includes Each registered host gets a dedicated dashboard page with: CPU user, system, and I/O wait utilization Memory use Swap use Load averages Disk use and read/write throughput Network inbound and outbound throughput A human-readable OS description for account owners

If you just want the recommendation: build the uptime poller yourself, put exponential backoff with jitter in front of every health check, and treat a 429 as a scheduling signal instead of an error you swallow. Query-style observability APIs hand you metrics and logs, not threshold rules or notification channels, so the polling worker is the thing that has to decide what "down" means and who gets woken up. That decision is the whole job. I got burned by exactly this. What follows is the pattern that survived the postmortem, the alternatives I weighed before writing a line of it, and the conditions where you should not do any of this yourself. The 429 my retry loop ate for six hours Last spring I was running a homegrown health checker for 40 internal services. One goroutine per service, all driven off the same 15-second ticker, which meant every check landed inside the same 200ms window. The status API we polled had a per-minute quota I'd never bothered to read, and for months it didn't matter, because 40 checks a minute sat comfortably under the ceiling. Then a colleague onboarded 12 more services, we crossed the quota, and the API started answering with HTTP 429. My retry wrapper caught it, retried three times in a tight loop, and on the last attempt returned the previous cached result — which said healthy . It logged the rate limit at debug level. Nobody reads debug. Six hours. Green dashboard. Dead queue consumer. We found out when a customer asked where their export was. The consumer had died on an unrelated deploy, the checker never noticed, and when I finally restarted it the backlog got re-processed on top of a manual replay I'd already run — two customers got the same notification twice. Duplicate deliveries are the specific thing I lose sleep over, and I had caused a batch of them with a retry loop that was trying to be helpful. The postmortem produced one line I now paste into every runbook: a check that can't reach the API reports unknown, never healthy.

Your monitoring catches problems on everything except itself. Here is how an always-firing Watchdog alert plus an external heartbeat check turns silence into a signal, so you find out when your own alerting dies. TL;DR A monitoring system can't reliably monitor its own failure, so use a dead man's switch. Create an always-firing Prometheus Watchdog alert and route it to an independent external heartbeat service. As long as the monitoring pipeline is working, the Watchdog continuously refreshes the heartbeat. If Prometheus, Alertmanager, or the delivery path fails, the heartbeat stops and the external service alerts you through a separate channel. The key is independence: the system responsible for detecting that your monitoring is down must not depend on the monitoring stack itself. One of the traps of creating alerts on a monitoring stack is the hidden assumption that the mechanism evaluating the alert is running properly and has the ability to evaluate it. Prometheus watches your hosts and Alertmanager delivers the warnings. But what watches Prometheus? If something goes wrong and the monitoring stack fails in the middle of the night, no alerts are going out but there is definitely a problem. That is the failure mode that you should be most concerned about, because it is the one your monitoring cannot report on. The fix is an old idea with a grim name: a dead man's switch. A train's dead man's switch stops the train when the operator stops holding it down. The safe state requires continuous positive action, while the absence of that action is what triggers the response. Applied to monitoring, it means building one alert whose silence is itself the alarm. Step one: an alert that always fires This feels backwards the first time you see it, and it took me a little time to get it right. Basically, you create an alert with a condition that is always true, so it fires constantly, forever, on purpose. In the Prometheus world this is conventionally called Watchdog. - aler

If you run more than a handful of OpenTelemetry Collectors, you already know the pain: a config change means SSHing into boxes, redeploying DaemonSets, or babysitting a Git pipeline per cluster, and you never quite trust that every agent is running the config you think it is. OpAMP fixes exactly that. It is a protocol that lets a central server push configuration to a fleet of Collectors, watch their health, and roll changes out in stages, without you touching each host. This post walks through how OpAMP works, the two ways a Collector can speak it, and the config you need to wire one up. The problem OpAMP solves A single Collector is easy. A hundred of them, spread across clusters, VMs, and edge nodes, is a fleet-management problem that has nothing to do with telemetry itself. Every observability team eventually builds some version of the same thing: a way to ship a new pipeline config, confirm it actually applied, and back it out when a processor starts dropping spans. Without a management protocol you end up gluing that together from ConfigMaps, Ansible runs, and dashboards that only tell you an agent is alive, not what config it is actually running. Config drift creeps in. One node keeps an old sampling rate for months because its rollout quietly failed and nobody noticed. OpAMP, the Open Agent Management Protocol, is the OpenTelemetry answer to this. Splunk donated it to the project in 2022, and it has since become the standard control channel for the Collector. It is worth pairing with a clear-eyed view of what a Collector actually is versus lighter agents; the OpenTelemetry Collector vs Grafana Alloy comparison covers that trade-off if you are still choosing a data plane. What OpAMP actually is OpAMP is a client/server network protocol for remote management of large fleets of data-collection agents. It is transport-flexible: agents connect to the server over either plain HTTP or a WebSocket, and the WebSocket path gives you a persistent bidirectional channel

I built an AI agent system that automatically switches to a backup AI model if the main one fails. I connected every step to SigNoz so I could track requests, monitor performance, and detect failures. I also built a diagnostic agent that reads the monitoring data and explains the reason for failures in simple language. During testing, it successfully detected a real AI provider outage and identified the root cause automatically. signoz

"Night gathers, and now my watch begins." The Night's Watch didn't exist to fight wars nobody saw coming — they existed because someone had to actually stand on the Wall and notice when something crossed it. That's the exact problem I kept running into with AI agents, and it's why I built Nights Watch for the "Agents of SigNoz" hackathon: a runtime resilience layer that catches an agent quietly drifting off its plan, explains why, and recovers — automatically. The problem nobody's watching for Most agent failures aren't dramatic. An agent doesn't crash, it doesn't throw an exception, it doesn't get flagged by a content filter. It just... does something slightly different from what it was asked. Told to "find and book a flight under $400," a subtly-drifted agent might reason its way into a $1,200 upgrade and report back "done" — technically true, catastrophically wrong. Nothing in a normal observability stack notices this, because nothing failed . The agent succeeded at the wrong thing. I wanted a system where SigNoz wasn't just a dashboard you check after something breaks — where it actively fed a decision-making loop while the agent was still running . Architecture, in one rule Everything else in the project falls out of one non-negotiable decision I made on day one: rollback state has to be local and durable, never dependent on an external service being reachable. If your resilience system's own safety net depends on a third-party API being up, you haven't built resilience, you've built a second point of failure. So the split looks like this: Local, critical path (SQLite): the Checkpoint Manager. Every agent step writes a durable checkpoint — plan, budget consumed, completed steps — to disk via Node's built-in node:sqlite . Rollback reads from here, always, no exceptions. SigNoz, decision-support only: the Policy Engine queries SigNoz's Query API for prior-run context before scoring severity, and the Explanation Layer calls SigNoz's MCP server to ground its natura

Evidence First, Answer Second: Building an Observable Industrial AI Agent with SigNoz Most AI systems are designed to give an answer. That is useful in a chatbot. On a factory floor, it can be dangerous. While building Industrial IoT Anomaly Control , I kept coming back to one question: What should an AI agent do when it detects a real problem but does not have enough evidence to explain the cause safely? My answer was simple: it should stop, show what it knows, and send the case to a human. This project is a real-time monitoring system for a simulated water-treatment plant. It streams live sensor data from six industrial assets, detects unusual behaviour, searches a knowledge base for similar incidents, and then chooses between two paths: recommend a safe action when the evidence is strong; escalate for human review when the evidence is weak. SigNoz is what makes this decision process visible. Instead of seeing only the final AI response, I can inspect the full path from the incoming sensor reading to anomaly detection, knowledge retrieval, policy checks, agent explanation, and recovery. The problem with traditional alarms Factories already collect large amounts of telemetry such as vibration, temperature, humidity, sequence numbers, and timestamps. The problem is not missing data. The problem is turning that data into a useful decision. A traditional threshold alarm may say: Vibration is above the configured limit. That still leaves the operator with several questions: Is the machine actually failing? Is the sensor or gateway sending bad data? Has this pattern happened before? Is there enough evidence to recommend maintenance? Why did the AI reach this conclusion? Repeated threshold alerts can also create alarm fatigue. If one fault produces dozens of alerts, operators may start treating them as noise. I wanted the system to create one investigation instead of another flood of alarms. What I built The demo represents six assets in a water-treatment plant. A TCP si

I keep running into (and hearing about) a specific kind of bug that never throws an error — an API you depend on quietly changes its response shape. A field disappears. A number becomes a string. Something that was always present is suddenly null. Nothing crashes immediately. It just produces wrong or missing data somewhere downstream, and you find out from a bug report, not a log. I'm curious how common this actually is outside my own experience, so — genuine question, not a pitch: Has this happened to you, with a third-party API or even an internal one your own team owns? How did you find out it happened — a user report, a stack trace somewhere unrelated, manual debugging? Do you currently do anything to catch this kind of thing before it bites you (contract tests, monitoring, or just... hoping)? If you don't do anything about it today, is that because it's not painful enough to bother, or because you just haven't found a lightweight way to? Not selling anything here, just trying to understand how real and how painful this actually is for people building on top of APIs day to day. Would genuinely appreciate hearing your experience, even a one-line "yeah this happened to me once, wasn't a big deal" is useful data.

I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post. Between 21 and 24 July it did that four times: 23 Jul Gemini 3.5 Flash -11.4 pts 24 Jul Gemini 3.1 Pro -2.9 pts 21 Jul Grok 4.3 -5.7 pts 22 Jul Llama 3.3 70B -2.9 pts Four regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong. None of those models got worse. Here's how I know, because the how is the only part worth reading. Two of them weren't the model Every point on the board carries a second number next to accuracy: reliability , the share of probe calls that actually came back. Look at the two Google alerts with that column showing: gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000 23 Jul acc 0.886 reliability 0.914 <- "-11.4 pts" gemini-3.1-pro 22 Jul acc 0.914 reliability 0.943 23 Jul acc 0.886 reliability 0.914 <- "-2.9 pts" 24 Jul acc 0.971 reliability 1.000 <- next clean run Accuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one. I know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight: api.groq.com -> 429: Rate limit reached for model ` llama-3.3-70b-versatile ` service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30 34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch. Gemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, higher than before the "regression." The other two were one question The remaining two alerts ar

You've probably seen this: Kubernetes probes are green, /health returns 200, CPU isn't on fire - and users are timing out. Process is up. Just not doing anything useful. Classic event loop freeze. Something sync and expensive (or a dumb busy loop) grabs the thread, and suddenly timers, HTTP, DB callbacks - all of it waits. Including your health endpoint, because that also needs the loop. monitorEventLoopDelay() is fine for dashboards. It won't yell at you while the process is stuck, and it definitely won't write a log line from outside the frozen loop. That's the part that annoyed me enough to build this. I made @js-ak/watchdog - tiny N-API addon. The monitor runs in C++ on its own thread and writes JSON lines to stderr/file even while the JS event loop is stuck. Your freeze / recovered handlers only run after the loop unblocks. Optional RSS/CPU, optional stack sample. npm install @js-ak/watchdog const watchdog = require("@js-ak/watchdog"); watchdog.on("freeze", (e) => console.log(e.event, e.duration_ms)); watchdog.on("recovered", (e) => console.log("back after", e.duration_ms, "ms")); watchdog.start({ freezeThresholdMs: 1000, logTarget: "stderr", // or "file" | "both" source: "payments-api", }); Real freezes usually aren't while (true) . More like giant JSON.parse, a cursed regex, sync crypto/compress, or some dependency doing sync I/O on a hot path. Stuff you only notice after people complain. Prebuilds for the usual platforms (Node 22+). MIT. Repo: https://github.com/JS-AK/watchdog Curious how other people catch loop stalls in prod - and what you'd actually want in the freeze payload. Roast the API if it's wrong.

A SQLite service can keep answering health checks while its WAL grows without a successful checkpoint. One forgotten read transaction is enough to reproduce the risk. Run a controlled drill: Enable WAL mode and create a small table. Open connection A, begin a read transaction, and keep it open. From connection B, commit batches of writes for 60 seconds. Sample WAL bytes and checkpoint results every second. Release A and observe recovery. PRAGMA journal_mode = WAL ; PRAGMA wal_checkpoint ( PASSIVE ); Record the three checkpoint counters plus duration, WAL file size, oldest transaction age, write latency, and free disk bytes. A green SELECT 1 says nothing about checkpoint progress. My fault-injection gate looks like this: Given: one reader holds its transaction When: 10,000 writes commit Then: writes remain bounded And: WAL growth triggers an alert before the disk budget When: the reader closes Then: checkpoint progress resumes and WAL size converges Do not start with an automatic TRUNCATE loop. A busy result is evidence of contention; aggressive checkpoints can add latency without fixing the reader lifecycle. First identify long transactions, ensure result iterators close, and define a maximum transaction age. SQLite’s WAL documentation explains that a checkpoint must stop when it reaches pages beyond an active reader’s end mark. This is why the drill needs concurrency rather than a synthetic file-growth assertion. Operational thresholds should be tied to a disk budget: alert when projected WAL growth reaches the remaining safe window, not at an arbitrary file size. The runbook should name how to find the oldest reader, how to shed writes, and when a restart is safer than waiting. A health check is useful only when it measures the subsystem that is failing.

The "Obvious" Math That's Wrong Engineer A: "Datadog is $15K/month. Prometheus is free. We should self-host." Engineer B: "But we'd need to pay an SRE to run it. That's $150K/year." Engineer A: "Prometheus doesn't need a full SRE. It's easy." Engineer B: "Famous last words." This conversation happens at every company. Both sides have points. The real math is more complex. The Total Cost Breakdown Managed (Datadog, New Relic, Dynatrace) : Licensing: $X/month (scales with hosts, events, logs) Integration time: 1-2 weeks per service Training: 1 day per new hire Ongoing: minimal Self-hosted (Prometheus + Grafana + Loki + Alertmanager) : Infrastructure: hosting costs (~$500-$5000/month depending on scale) Initial setup: 2-4 weeks of engineering time Ongoing maintenance: 10-20% of 1 FTE Upgrade costs: quarterly, each upgrade ~1 week Storage growth: ~20% per year Expertise: junior → senior SRE hire required The honest answer: managed is cheaper for teams under 50 engineers. Self-hosted becomes cheaper around 200+ engineers if you can run it well . The Real Variables It's not just licensing cost vs. hosting cost. These factors matter more: 1. Data volume growth Managed tools charge per GB ingested or per metric. If your logs 10x, your bill 10x's. Self-hosted scales linearly with compute. You control the growth. 2. Retention requirements Managed tools often charge extra for long retention. Self-hosted you store as much as your disk allows. 3. Cardinality Prometheus dies at high cardinality. Datadog handles it but charges more. High-cardinality metrics are where self-hosted breaks. 4. Incident rate Heavy incident load means heavy query load on your monitoring tools. Self-hosted needs bigger compute for this. 5. Team expertise If your team has never run Prometheus, you'll spend 6 months in the pit learning cardinality mistakes, retention tuning, and HA setups. That's not free. The Break-Even Calculation Rough calculation for a 50-engineer startup: Managed (Datadog) : - Licensi

The most expensive thing you can do with an AI agent is watch it. Not audit it. Not review its output. Watch it -- step by step, approval by approval, second-guessing every action before it takes the next one. And yet that is precisely how most engineering teams are deploying AI agents in 2026: on a leash so short the agent cannot take three steps without a human tapping it on the shoulder. I understand why. The models hallucinate. The stakes are real. Nobody wants to be the engineering manager who let an AI agent push a bad migration to production at 2am. So we wrap the agents in confirmation dialogs, require human sign-off at every branch point, and celebrate our careful governance. What we have actually built is an automation system that requires more human attention than the manual process it replaced. The better answer is not more control at the action level. It is better design at the intent level. Steer by intent, monitor by exception. Tell the agent clearly what outcome you need, what it must never do, and what constitutes a result worth stopping for. Then let it work. Watch the outcomes, not the steps. We have built automation systems that require more human attention than the manual process they replaced. That is not a governance success. That is a design failure. Why we got here The model for human-AI collaboration that most teams are using today was inherited from the model for junior developer supervision. You review every pull request. You approve every deployment. You sign off on every schema change. That model exists because junior developers are learning, because their mental models are incomplete, because their judgment has not yet been earned. Applied to AI agents, it assumes the same thing: the agent is a novice that needs supervision. But an AI agent is not a junior developer. It does not have an incomplete mental model of the codebase that will improve with mentorship. It has exactly the mental model you gave it via its context, its tools, and

Today I'm excited to launch RegionCheck , a tool for testing, monitoring, and debugging endpoints from cloud regions around the world. The idea is simple - test an endpoint's DNS and HTTP connectivity and response from a defined set of cloud regions. Whether you're troubleshooting an API, validating a deployment, checking DNS propagation, or investigating latency, seeing the results from multiple cloud regions can quickly reveal issues that aren't obvious from your own machine. What is RegionCheck? RegionCheck lets you run endpoint checks from AWS, Azure, and Google Cloud regions without provisioning infrastructure or maintaining test instances. Current capabilities include: HTTP endpoint testing DNS lookups TLS certificate validation Continuous monitoring with alerts Side-by-side comparison across cloud providers and regions Shareable result pages for collaboration API/MCP access for automation and agents Why I built it When debugging production issues, I often wanted to answer questions like: Is DNS returning the same result everywhere? Or is geo-DNS returning the results intended? Is TLS certificate propagation for my CDN working as intended? Is one region significantly slower than another? Is my CDN caching working as expected? Are my geo-HTTP redirects working as intended? (For some interesting examples try www.yahoo.com and www.cnn.com in non-US regions) There are many tools that exist that provide these answers, but nothing that answers all of these questions in one place. That's what RegionCheck aims to provide. Who it's for RegionCheck is designed for engineers who work with cloud infrastructure, including: DevOps engineers Site Reliability Engineers (SREs) Platform engineers Backend developers Anyone who likes to take a peek at backend infrastructure Try it out RegionCheck is available at https://regioncheck.io You can run free checks directly from the website; or create an account to access monitoring, alerting, the API, and additional features. I'd love

A 90% discount doesn't come from nowhere. Someone pays for it, and usually that someone is you: with your data, with model quality, or with somebody's stolen account. An entire industry has grown up around Anthropic and OpenAI over there. They call them "transfer stations": proxies that resell tokens from Western models. This stopped being a garage workaround for rate limits a long time ago. It's a market with its own economics, and it runs on a few things. Accounts. These services mass-produce free and Max subscriptions through stand-in registrants and other people's KYC, then wrap them in a pseudo-API. Model swapping. You pay for the flagship, but your simple request gets quietly routed to something cheaper. Who actually generated the answer? You can't check. Data. The proxy operator sees all of it: prompts, responses, code, internal context. Even if they never sell the logs, you don't know where those logs sit, who reads them, or whether they ended up in somebody else's training run. Take Yunwu. By their own price list, access to Anthropic models runs up to 93% below official. The price alone proves nothing. But the market logic is simple: restrictions don't kill demand, they make it more expensive, and they feed the workaround industry. In June 2026 Anthropic accused entities tied to Alibaba/Qwen of a campaign to siphon Claude: nearly 25,000 fake accounts and 28.8 million requests in six weeks, to distill their own models. DeepSeek, Moonshot and MiniMax have caught similar accusations before. Same grey access, just wholesale. But the problem is bigger than China. Even with fully legitimate access, most teams can't answer three questions about their own production: which model actually served the request, what it cost, and whether quality dropped. The grey market just takes that blindness to its logical end. I think the priorities here are upside down. For a serious product, production visibility and control over your data matter more than a few percent saved on

Most AI incident diagnosis tools will happily produce a root cause even when the evidence is weak. Argus takes a different approach. When an anomaly fires, Argus runs five independent diagnoses against the same incident window. If they converge on the same root cause, it returns a confident diagnosis. If they don't, it returns novel instead of pretending it knows the answer. It's a single Go binary. The first version had Kafka, microservices, and two databases. It looked impressive on paper, but nobody would actually run it. I tore it down into a single process and replaced Kafka with an in-process event bus. Run it with docker run, bring your own Anthropic API key, and your telemetry never leaves the box. It ingests OTLP or Prometheus remote_write; point your telemetry to a single endpoint. I've validated it on synthetic cases, reconstructed real postmortems (Cloudflare 2019/2022), and my own distributed system. It hasn't yet been tested against messy real-world production telemetry, which is exactly the kind of feedback I'm looking for. GitHub: https://github.com/k1ngalph0x/argus I'd genuinely appreciate people trying it out and telling me where the design falls apart, what feels over-engineered, or what you'd change.

TL;DR: If you only collect metrics, Prometheus Agent mode is lightweight, familiar, and difficult to beat. If you collect metrics, logs, or traces together, or expect to in the future, Grafana Alloy's unified pipeline is usually worth the additional complexity. Once you've decided to move from pull-based scraping to a push architecture , the next question is which agent should actually run on each host. In 2026, the two strongest choices are Prometheus Agent mode and Grafana Alloy. I run Alloy across my production fleet, but that doesn't automatically make it the right answer for everyone. The Shift in the Monitoring Landscape Over the last couple of years, Grafana has consolidated both metrics and log collection into Grafana Alloy. Grafana Agent reached end of life on November 1, 2025, and Promtail followed on March 2, 2026. Neither receives security fixes anymore. The practical choice moving forward: Feature Prometheus Agent Grafana Alloy Metrics ✅ ✅ Logs ❌ ✅ Traces ❌ ✅ Config Prometheus YAML Alloy components Footprint Smaller Larger Learning curve Low Moderate Future direction Metrics agent Unified telemetry The table gives the short answer. The rest of this article explains where those differences actually matter in practice. Prometheus Agent mode. Run the Prometheus binary with the --agent flag and it stops acting as a full Prometheus server. It no longer stores local TSDB blocks, evaluates alerting rules, or serves queries. Instead, it scrapes targets, buffers samples in a write-ahead log, and forwards them upstream via remote_write . It is Prometheus with the storage and query layers removed. Grafana Alloy. A single agent that collects metrics, logs, and traces, processes them in a component pipeline, and pushes each signal to its backend. It embeds many exporters directly, so a line like prometheus.exporter.unix "node_exporter" {} gives you full node_exporter functionality without installing a separate binary. The Case for Prometheus Agent If you only need m

Every time I shipped something, the same thought hit me a few hours later: It works on my machine. It works in staging. But is it actually working for the people using it right now? I had analytics. I had a green dashboard. And I still had no honest answer to that question. Users would quietly leave, a button would silently break on Safari, a page would crawl on a mid-range Android, and I'd find out days later, if at all. That gap is what I ended up building HeronSignal to close. But before I talk about the tool, let me talk about the pain, because I think you've felt at least one version of it. The pain, depending on who you are If you're a vibe coder / solo builder You ship fast. Cursor, Claude, v0, a Vercel deploy, and it's live. Beautiful. Then… nothing. You have no idea what happens after "Deploy successful." Is the checkout button throwing an error on mobile? Is your landing page slow enough that half your visitors bounce before it paints? You don't know, because setting up "real" monitoring feels like a second job: a Datadog dashboard you'll never look at, a Sentry config you half-finish. So you just… hope. And hope is not a monitoring strategy. If you're an engineer Your problem isn't no data. It's too much . Ten dashboards, alert fatigue, a Sentry inbox with 400 issues where 390 are noise. Something's clearly wrong, but which thing actually matters? You spend your morning triaging instead of fixing. And when you finally pick an error, you get a stack trace with zero context: no idea what page it happened on, what the user was doing, or how to reproduce it. Triage is not the job. Fixing is the job. But the tools make you do the triage first. If you're a product person You can see in your funnel that people drop off at step 3. What you can't see is why . Was it a JS error? A slow page? A confusing layout? Your analytics tool tells you what happened but never why , and the engineering dashboards that might explain it are unreadable walls of numbers. So you gue

If you ask me what part of the system I protect the most, the answer is the database. I've been writing software alone for twenty-four years, and across every platform I've built, the rule has stayed the same: the web servers can take whatever you throw at them, the batches can be rebuilt, but the database has to stay idle on purpose. Not because I love idle databases, but because the day a database actually starts to struggle is a day with very few good options. This article is about what "keep the database idle on purpose" actually means in practice, and about one particular kind of graph that, in my experience, almost nobody is watching. The three layers and what each of them gets I think of a production system as having three tiers, and each tier gets a different rule. The web server tier can be horizontally scaled. If load grows, you add machines. If something is wrong, you take a machine out of the pool, and the others handle it. Failures here are visible immediately, and they're cheap to recover from. The batch server tier can be scaled up or out depending on the work. A batch that's too slow can be split. A batch that crashes can be retried. End users don't see batch servers, so a stuck batch is a problem for me and not for them. Some headroom up here is fine. The database tier is the one I treat completely differently. The database is not where you absorb load. The database is what you protect from load. The reason is simple: the other tiers can be rebuilt or re-scaled. The database is the irreplaceable record. If it slows down, everything slows down. If it falls over, you don't have many minutes before the rest of the stack notices. So my rule for the database is: keep it idle. Not idle in the sense of "doing nothing." Idle in the sense of "running well below its capacity, at all times, so that any extra load it picks up has somewhere to go." For more than a decade I ran a large appliance-grade database where I kept the load average below 1 at all times. N

Understanding Observability with the LGTM Stack From "what happened last night?" to "here's exactly what happened and why" — in under 5 minutes Table of Contents Introduction What Is Observability? The Three Pillars of Observability Metrics Logs Traces Why You Need All Three Together The LGTM Stack Architecture: How It All Fits Together OpenTelemetry: The Instrumentation Standard The OTel Collector: The Brain of the Pipeline Loki: Log Aggregation Tempo: Distributed Tracing Mimir: Metrics at Scale Grafana: Connecting the Dots Conclusion Introduction Let me tell you a story that probably sounds familiar. It's 2 AM on a Sunday. Your API is slow. Users are complaining. But you're not at your desk — you're in a Sleeping, or just living your life. You have no idea it's even happening. The next morning you walk into the office and your boss meets you at the door. "Hey, the API was really slow yesterday around 2 AM. What happened?" And you're stuck. Completely stuck. You pull up the server logs — it's a wall of unformatted text. Maybe the issue already fixed itself. Maybe the container restarted overnight and the logs are gone. You weren't there, and your system left no trail. So you say the thing every developer dreads saying: "I don't know. I'll look into it." Now imagine the exact same situation — but this time you have observability set up. You open your dashboard, set the time range to yesterday 2 AM, and within two minutes you can see everything. Response times spiked to 4 seconds. The database connection pool got exhausted. And it started the exact moment a scheduled batch job kicked off and hammered the DB with hundreds of queries at once. You have a graph. You have traces. You have the exact log line that caused it. You walk back to your boss with your laptop: "Here's what happened and here's the fix." That's observability. Your system tells its own story — even when you're not watching. That's what this blog is about. I'll walk you through what observability actua