标签：#monitoring

I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post. Between 21 and 24 July it did that four times: 23 Jul Gemini 3.5 Flash -11.4 pts 24 Jul Gemini 3.1 Pro -2.9 pts 21 Jul Grok 4.3 -5.7 pts 22 Jul Llama 3.3 70B -2.9 pts Four regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong. None of those models got worse. Here's how I know, because the how is the only part worth reading. Two of them weren't the model Every point on the board carries a second number next to accuracy: reliability , the share of probe calls that actually came back. Look at the two Google alerts with that column showing: gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000 23 Jul acc 0.886 reliability 0.914 <- "-11.4 pts" gemini-3.1-pro 22 Jul acc 0.914 reliability 0.943 23 Jul acc 0.886 reliability 0.914 <- "-2.9 pts" 24 Jul acc 0.971 reliability 1.000 <- next clean run Accuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one. I know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight: api.groq.com -> 429: Rate limit reached for model ` llama-3.3-70b-versatile ` service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30 34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch. Gemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, higher than before the "regression." The other two were one question The remaining two alerts ar

You've probably seen this: Kubernetes probes are green, /health returns 200, CPU isn't on fire - and users are timing out. Process is up. Just not doing anything useful. Classic event loop freeze. Something sync and expensive (or a dumb busy loop) grabs the thread, and suddenly timers, HTTP, DB callbacks - all of it waits. Including your health endpoint, because that also needs the loop. monitorEventLoopDelay() is fine for dashboards. It won't yell at you while the process is stuck, and it definitely won't write a log line from outside the frozen loop. That's the part that annoyed me enough to build this. I made @js-ak/watchdog - tiny N-API addon. The monitor runs in C++ on its own thread and writes JSON lines to stderr/file even while the JS event loop is stuck. Your freeze / recovered handlers only run after the loop unblocks. Optional RSS/CPU, optional stack sample. npm install @js-ak/watchdog const watchdog = require("@js-ak/watchdog"); watchdog.on("freeze", (e) => console.log(e.event, e.duration_ms)); watchdog.on("recovered", (e) => console.log("back after", e.duration_ms, "ms")); watchdog.start({ freezeThresholdMs: 1000, logTarget: "stderr", // or "file" | "both" source: "payments-api", }); Real freezes usually aren't while (true) . More like giant JSON.parse, a cursed regex, sync crypto/compress, or some dependency doing sync I/O on a hot path. Stuff you only notice after people complain. Prebuilds for the usual platforms (Node 22+). MIT. Repo: https://github.com/JS-AK/watchdog Curious how other people catch loop stalls in prod - and what you'd actually want in the freeze payload. Roast the API if it's wrong.

A SQLite service can keep answering health checks while its WAL grows without a successful checkpoint. One forgotten read transaction is enough to reproduce the risk. Run a controlled drill: Enable WAL mode and create a small table. Open connection A, begin a read transaction, and keep it open. From connection B, commit batches of writes for 60 seconds. Sample WAL bytes and checkpoint results every second. Release A and observe recovery. PRAGMA journal_mode = WAL ; PRAGMA wal_checkpoint ( PASSIVE ); Record the three checkpoint counters plus duration, WAL file size, oldest transaction age, write latency, and free disk bytes. A green SELECT 1 says nothing about checkpoint progress. My fault-injection gate looks like this: Given: one reader holds its transaction When: 10,000 writes commit Then: writes remain bounded And: WAL growth triggers an alert before the disk budget When: the reader closes Then: checkpoint progress resumes and WAL size converges Do not start with an automatic TRUNCATE loop. A busy result is evidence of contention; aggressive checkpoints can add latency without fixing the reader lifecycle. First identify long transactions, ensure result iterators close, and define a maximum transaction age. SQLite’s WAL documentation explains that a checkpoint must stop when it reaches pages beyond an active reader’s end mark. This is why the drill needs concurrency rather than a synthetic file-growth assertion. Operational thresholds should be tied to a disk budget: alert when projected WAL growth reaches the remaining safe window, not at an arbitrary file size. The runbook should name how to find the oldest reader, how to shed writes, and when a restart is safer than waiting. A health check is useful only when it measures the subsystem that is failing.

The "Obvious" Math That's Wrong Engineer A: "Datadog is $15K/month. Prometheus is free. We should self-host." Engineer B: "But we'd need to pay an SRE to run it. That's $150K/year." Engineer A: "Prometheus doesn't need a full SRE. It's easy." Engineer B: "Famous last words." This conversation happens at every company. Both sides have points. The real math is more complex. The Total Cost Breakdown Managed (Datadog, New Relic, Dynatrace) : Licensing: $X/month (scales with hosts, events, logs) Integration time: 1-2 weeks per service Training: 1 day per new hire Ongoing: minimal Self-hosted (Prometheus + Grafana + Loki + Alertmanager) : Infrastructure: hosting costs (~$500-$5000/month depending on scale) Initial setup: 2-4 weeks of engineering time Ongoing maintenance: 10-20% of 1 FTE Upgrade costs: quarterly, each upgrade ~1 week Storage growth: ~20% per year Expertise: junior → senior SRE hire required The honest answer: managed is cheaper for teams under 50 engineers. Self-hosted becomes cheaper around 200+ engineers if you can run it well . The Real Variables It's not just licensing cost vs. hosting cost. These factors matter more: 1. Data volume growth Managed tools charge per GB ingested or per metric. If your logs 10x, your bill 10x's. Self-hosted scales linearly with compute. You control the growth. 2. Retention requirements Managed tools often charge extra for long retention. Self-hosted you store as much as your disk allows. 3. Cardinality Prometheus dies at high cardinality. Datadog handles it but charges more. High-cardinality metrics are where self-hosted breaks. 4. Incident rate Heavy incident load means heavy query load on your monitoring tools. Self-hosted needs bigger compute for this. 5. Team expertise If your team has never run Prometheus, you'll spend 6 months in the pit learning cardinality mistakes, retention tuning, and HA setups. That's not free. The Break-Even Calculation Rough calculation for a 50-engineer startup: Managed (Datadog) : - Licensi

The most expensive thing you can do with an AI agent is watch it. Not audit it. Not review its output. Watch it -- step by step, approval by approval, second-guessing every action before it takes the next one. And yet that is precisely how most engineering teams are deploying AI agents in 2026: on a leash so short the agent cannot take three steps without a human tapping it on the shoulder. I understand why. The models hallucinate. The stakes are real. Nobody wants to be the engineering manager who let an AI agent push a bad migration to production at 2am. So we wrap the agents in confirmation dialogs, require human sign-off at every branch point, and celebrate our careful governance. What we have actually built is an automation system that requires more human attention than the manual process it replaced. The better answer is not more control at the action level. It is better design at the intent level. Steer by intent, monitor by exception. Tell the agent clearly what outcome you need, what it must never do, and what constitutes a result worth stopping for. Then let it work. Watch the outcomes, not the steps. We have built automation systems that require more human attention than the manual process they replaced. That is not a governance success. That is a design failure. Why we got here The model for human-AI collaboration that most teams are using today was inherited from the model for junior developer supervision. You review every pull request. You approve every deployment. You sign off on every schema change. That model exists because junior developers are learning, because their mental models are incomplete, because their judgment has not yet been earned. Applied to AI agents, it assumes the same thing: the agent is a novice that needs supervision. But an AI agent is not a junior developer. It does not have an incomplete mental model of the codebase that will improve with mentorship. It has exactly the mental model you gave it via its context, its tools, and

Today I'm excited to launch RegionCheck , a tool for testing, monitoring, and debugging endpoints from cloud regions around the world. The idea is simple - test an endpoint's DNS and HTTP connectivity and response from a defined set of cloud regions. Whether you're troubleshooting an API, validating a deployment, checking DNS propagation, or investigating latency, seeing the results from multiple cloud regions can quickly reveal issues that aren't obvious from your own machine. What is RegionCheck? RegionCheck lets you run endpoint checks from AWS, Azure, and Google Cloud regions without provisioning infrastructure or maintaining test instances. Current capabilities include: HTTP endpoint testing DNS lookups TLS certificate validation Continuous monitoring with alerts Side-by-side comparison across cloud providers and regions Shareable result pages for collaboration API/MCP access for automation and agents Why I built it When debugging production issues, I often wanted to answer questions like: Is DNS returning the same result everywhere? Or is geo-DNS returning the results intended? Is TLS certificate propagation for my CDN working as intended? Is one region significantly slower than another? Is my CDN caching working as expected? Are my geo-HTTP redirects working as intended? (For some interesting examples try www.yahoo.com and www.cnn.com in non-US regions) There are many tools that exist that provide these answers, but nothing that answers all of these questions in one place. That's what RegionCheck aims to provide. Who it's for RegionCheck is designed for engineers who work with cloud infrastructure, including: DevOps engineers Site Reliability Engineers (SREs) Platform engineers Backend developers Anyone who likes to take a peek at backend infrastructure Try it out RegionCheck is available at https://regioncheck.io You can run free checks directly from the website; or create an account to access monitoring, alerting, the API, and additional features. I'd love

A 90% discount doesn't come from nowhere. Someone pays for it, and usually that someone is you: with your data, with model quality, or with somebody's stolen account. An entire industry has grown up around Anthropic and OpenAI over there. They call them "transfer stations": proxies that resell tokens from Western models. This stopped being a garage workaround for rate limits a long time ago. It's a market with its own economics, and it runs on a few things. Accounts. These services mass-produce free and Max subscriptions through stand-in registrants and other people's KYC, then wrap them in a pseudo-API. Model swapping. You pay for the flagship, but your simple request gets quietly routed to something cheaper. Who actually generated the answer? You can't check. Data. The proxy operator sees all of it: prompts, responses, code, internal context. Even if they never sell the logs, you don't know where those logs sit, who reads them, or whether they ended up in somebody else's training run. Take Yunwu. By their own price list, access to Anthropic models runs up to 93% below official. The price alone proves nothing. But the market logic is simple: restrictions don't kill demand, they make it more expensive, and they feed the workaround industry. In June 2026 Anthropic accused entities tied to Alibaba/Qwen of a campaign to siphon Claude: nearly 25,000 fake accounts and 28.8 million requests in six weeks, to distill their own models. DeepSeek, Moonshot and MiniMax have caught similar accusations before. Same grey access, just wholesale. But the problem is bigger than China. Even with fully legitimate access, most teams can't answer three questions about their own production: which model actually served the request, what it cost, and whether quality dropped. The grey market just takes that blindness to its logical end. I think the priorities here are upside down. For a serious product, production visibility and control over your data matter more than a few percent saved on

Most AI incident diagnosis tools will happily produce a root cause even when the evidence is weak. Argus takes a different approach. When an anomaly fires, Argus runs five independent diagnoses against the same incident window. If they converge on the same root cause, it returns a confident diagnosis. If they don't, it returns novel instead of pretending it knows the answer. It's a single Go binary. The first version had Kafka, microservices, and two databases. It looked impressive on paper, but nobody would actually run it. I tore it down into a single process and replaced Kafka with an in-process event bus. Run it with docker run, bring your own Anthropic API key, and your telemetry never leaves the box. It ingests OTLP or Prometheus remote_write; point your telemetry to a single endpoint. I've validated it on synthetic cases, reconstructed real postmortems (Cloudflare 2019/2022), and my own distributed system. It hasn't yet been tested against messy real-world production telemetry, which is exactly the kind of feedback I'm looking for. GitHub: https://github.com/k1ngalph0x/argus I'd genuinely appreciate people trying it out and telling me where the design falls apart, what feels over-engineered, or what you'd change.

TL;DR: If you only collect metrics, Prometheus Agent mode is lightweight, familiar, and difficult to beat. If you collect metrics, logs, or traces together, or expect to in the future, Grafana Alloy's unified pipeline is usually worth the additional complexity. Once you've decided to move from pull-based scraping to a push architecture , the next question is which agent should actually run on each host. In 2026, the two strongest choices are Prometheus Agent mode and Grafana Alloy. I run Alloy across my production fleet, but that doesn't automatically make it the right answer for everyone. The Shift in the Monitoring Landscape Over the last couple of years, Grafana has consolidated both metrics and log collection into Grafana Alloy. Grafana Agent reached end of life on November 1, 2025, and Promtail followed on March 2, 2026. Neither receives security fixes anymore. The practical choice moving forward: Feature Prometheus Agent Grafana Alloy Metrics ✅ ✅ Logs ❌ ✅ Traces ❌ ✅ Config Prometheus YAML Alloy components Footprint Smaller Larger Learning curve Low Moderate Future direction Metrics agent Unified telemetry The table gives the short answer. The rest of this article explains where those differences actually matter in practice. Prometheus Agent mode. Run the Prometheus binary with the --agent flag and it stops acting as a full Prometheus server. It no longer stores local TSDB blocks, evaluates alerting rules, or serves queries. Instead, it scrapes targets, buffers samples in a write-ahead log, and forwards them upstream via remote_write . It is Prometheus with the storage and query layers removed. Grafana Alloy. A single agent that collects metrics, logs, and traces, processes them in a component pipeline, and pushes each signal to its backend. It embeds many exporters directly, so a line like prometheus.exporter.unix "node_exporter" {} gives you full node_exporter functionality without installing a separate binary. The Case for Prometheus Agent If you only need m

Every time I shipped something, the same thought hit me a few hours later: It works on my machine. It works in staging. But is it actually working for the people using it right now? I had analytics. I had a green dashboard. And I still had no honest answer to that question. Users would quietly leave, a button would silently break on Safari, a page would crawl on a mid-range Android, and I'd find out days later, if at all. That gap is what I ended up building HeronSignal to close. But before I talk about the tool, let me talk about the pain, because I think you've felt at least one version of it. The pain, depending on who you are If you're a vibe coder / solo builder You ship fast. Cursor, Claude, v0, a Vercel deploy, and it's live. Beautiful. Then… nothing. You have no idea what happens after "Deploy successful." Is the checkout button throwing an error on mobile? Is your landing page slow enough that half your visitors bounce before it paints? You don't know, because setting up "real" monitoring feels like a second job: a Datadog dashboard you'll never look at, a Sentry config you half-finish. So you just… hope. And hope is not a monitoring strategy. If you're an engineer Your problem isn't no data. It's too much . Ten dashboards, alert fatigue, a Sentry inbox with 400 issues where 390 are noise. Something's clearly wrong, but which thing actually matters? You spend your morning triaging instead of fixing. And when you finally pick an error, you get a stack trace with zero context: no idea what page it happened on, what the user was doing, or how to reproduce it. Triage is not the job. Fixing is the job. But the tools make you do the triage first. If you're a product person You can see in your funnel that people drop off at step 3. What you can't see is why . Was it a JS error? A slow page? A confusing layout? Your analytics tool tells you what happened but never why , and the engineering dashboards that might explain it are unreadable walls of numbers. So you gue

If you ask me what part of the system I protect the most, the answer is the database. I've been writing software alone for twenty-four years, and across every platform I've built, the rule has stayed the same: the web servers can take whatever you throw at them, the batches can be rebuilt, but the database has to stay idle on purpose. Not because I love idle databases, but because the day a database actually starts to struggle is a day with very few good options. This article is about what "keep the database idle on purpose" actually means in practice, and about one particular kind of graph that, in my experience, almost nobody is watching. The three layers and what each of them gets I think of a production system as having three tiers, and each tier gets a different rule. The web server tier can be horizontally scaled. If load grows, you add machines. If something is wrong, you take a machine out of the pool, and the others handle it. Failures here are visible immediately, and they're cheap to recover from. The batch server tier can be scaled up or out depending on the work. A batch that's too slow can be split. A batch that crashes can be retried. End users don't see batch servers, so a stuck batch is a problem for me and not for them. Some headroom up here is fine. The database tier is the one I treat completely differently. The database is not where you absorb load. The database is what you protect from load. The reason is simple: the other tiers can be rebuilt or re-scaled. The database is the irreplaceable record. If it slows down, everything slows down. If it falls over, you don't have many minutes before the rest of the stack notices. So my rule for the database is: keep it idle. Not idle in the sense of "doing nothing." Idle in the sense of "running well below its capacity, at all times, so that any extra load it picks up has somewhere to go." For more than a decade I ran a large appliance-grade database where I kept the load average below 1 at all times. N

Understanding Observability with the LGTM Stack From "what happened last night?" to "here's exactly what happened and why" — in under 5 minutes Table of Contents Introduction What Is Observability? The Three Pillars of Observability Metrics Logs Traces Why You Need All Three Together The LGTM Stack Architecture: How It All Fits Together OpenTelemetry: The Instrumentation Standard The OTel Collector: The Brain of the Pipeline Loki: Log Aggregation Tempo: Distributed Tracing Mimir: Metrics at Scale Grafana: Connecting the Dots Conclusion Introduction Let me tell you a story that probably sounds familiar. It's 2 AM on a Sunday. Your API is slow. Users are complaining. But you're not at your desk — you're in a Sleeping, or just living your life. You have no idea it's even happening. The next morning you walk into the office and your boss meets you at the door. "Hey, the API was really slow yesterday around 2 AM. What happened?" And you're stuck. Completely stuck. You pull up the server logs — it's a wall of unformatted text. Maybe the issue already fixed itself. Maybe the container restarted overnight and the logs are gone. You weren't there, and your system left no trail. So you say the thing every developer dreads saying: "I don't know. I'll look into it." Now imagine the exact same situation — but this time you have observability set up. You open your dashboard, set the time range to yesterday 2 AM, and within two minutes you can see everything. Response times spiked to 4 seconds. The database connection pool got exhausted. And it started the exact moment a scheduled batch job kicked off and hammered the DB with hundreds of queries at once. You have a graph. You have traces. You have the exact log line that caused it. You walk back to your boss with your laptop: "Here's what happened and here's the fix." That's observability. Your system tells its own story — even when you're not watching. That's what this blog is about. I'll walk you through what observability actua

RQ (Redis Queue) is a delightfully simple way to run background jobs in Python. That simplicity is also why teams under-monitor it: it just works, until a downstream API gets slow or a bad deploy ships, and jobs start failing in bulk — quietly. Here's what to watch and how to get alerted before a customer tells you. RQ failures don't announce themselves When a job raises, RQ moves it to the FailedJobRegistry and moves on. The worker keeps running; nothing crashes. If you're not looking at that registry, the failure is invisible — the same trap BullMQ, Celery, and every robust queue share. So the job is to reach into the queue's state and turn it into a signal. The four signals that matter for RQ Failure count / rate — jobs landing in the FailedJobRegistry over a window. Backlog — how many jobs are queued vs. being worked; is the worker keeping up? Latency — how long jobs take, and how long they wait before a worker picks them up. Worker liveness — are your workers actually alive and heartbeating? Where to read them RQ exposes queue and registry state directly: from redis import Redis from rq import Queue from rq.registry import FailedJobRegistry , StartedJobRegistry redis = Redis () q = Queue ( " default " , connection = redis ) queued = len ( q ) # backlog failed = FailedJobRegistry ( queue = q ) # failures started = StartedJobRegistry ( queue = q ) # in-flight print ( " queued: " , queued ) print ( " failed: " , len ( failed )) print ( " started: " , len ( started )) Poll this on an interval and store the series — a single snapshot hides the trend , which is the part that matters. For failures specifically, walk the registry to get the actual exceptions: for job_id in failed . get_job_ids (): job = q . fetch_job ( job_id ) print ( job . id , job . exc_info . splitlines ()[ - 1 ] if job . exc_info else "" ) Two gotchas: Group by exception, not by job. A thousand jobs failing with the same traceback is one incident. Normalize the message (strip IDs, timestamps, host

If you've been monitoring Linux servers for any length of time, there's a good chance node_exporter was the first thing you installed. It's lightweight, reliable, and exposes a huge amount of machine metrics for Prometheus to scrape. For years, it has been the default answer. As your infrastructure grows, though, your monitoring stack usually grows with it. First comes log collection. Then traces. Before long you're running node_exporter , a log shipper, and maybe another telemetry agent. Each component has its own configuration, service unit, upgrade cycle, and failure modes. Grafana Alloy changes that by consolidating those responsibilities into a single telemetry agent. This post walks through migrating from node_exporter to Alloy on a real fleet, one server at a time, while maintaining continuous visibility throughout the process. These are the exact steps that survived contact with production on the Irin monitoring stack, not the idealized version that looks clean in a diagram. TL;DR If you're already running node_exporter , don't replace it overnight. Install Grafana Alloy alongside it, configure Alloy's built-in prometheus.exporter.unix component, verify that metrics are reaching your remote Prometheus instance, and only then retire node_exporter. Migrating one server at a time minimizes risk, preserves visibility, and positions your infrastructure for logs, traces, and future telemetry without deploying additional agents. The real difference is the direction of travel Before getting started, it's worth understanding what actually changes. This isn't simply replacing one monitoring agent with another. node_exporter is a server. It listens on a port, typically 9100,and waits for Prometheus to connect and scrape metrics. That means every monitored machine needs an open endpoint, network connectivity from Prometheus, firewall rules, and scrape configurations. Alloy flips that model around. Instead of waiting for Prometheus to connect, Alloy collects metrics loca

HA dashboard cho Postgres replication: vì sao "có replica" không đồng nghĩa "đang HA", và cách để replica chết không âm thầm Streaming replication trong Postgres là một tunnel WAL từ primary sang standby: primary chạy một walsender cho mỗi standby, standby chạy một walreceiver nhận WAL và một startup process replay. Đau nhất trong vận hành không phải setup — mà là replica đã ngắt nhiều giờ mà không ai biết, tới lúc primary chết mới phát hiện HA thực ra là single-node từ tuần trước. HA dashboard là tập metric + alert được rút ra từ pg_stat_replication , pg_stat_wal_receiver , và pg_replication_slots để ép mọi trạng thái xấu — replica disconnect, replay đứng, slot pin WAL, sync standby biến mất — thành tín hiệu nhìn thấy được trước khi biến thành sự cố. Cơ chế hoạt động Trên primary, mỗi standby đang kết nối tạo ra một backend loại walsender — Postgres đọc WAL từ pg_wal/ (hoặc từ WAL buffers khi còn nóng) và stream qua replication connection. Trên standby, walreceiver nhận từng WAL record, ghi vào pg_wal/ local, fsync (tùy synchronous_commit ), rồi startup process apply record lên shared buffers — đây chính là replay. Bốn LSN xuất hiện trong luồng này và tương ứng với bốn cột trong pg_stat_replication : sent_lsn — byte cuối cùng primary đã gửi qua socket. write_lsn — byte cuối cùng standby đã write() vào OS page cache. flush_lsn — byte cuối cùng standby đã fsync xuống đĩa. replay_lsn — byte cuối cùng standby đã replay vào shared buffers (dữ liệu đã "thấy được" trên standby). Postgres docs quy định replay_lsn <= flush_lsn <= write_lsn <= sent_lsn <= pg_current_wal_lsn() — bốn "vạch" này chính là bốn nhịp của lag. Ba cột write_lag , flush_lag , replay_lag (kiểu interval ) là thời gian mà standby chậm hơn primary tương ứng với ba mốc write/flush/replay — được đo qua feedback message định kỳ từ standby. -- Trên primary: bức tranh đầy đủ cho một HA dashboard SELECT application_name , client_addr , state , -- streaming | catchup | startup | backup | stopping sync_state , --

Production lock analysis: vì sao pg_stat_activity một mình không đủ, và join với pg_locks mới ra root cause Lock contention trong Postgres hiếm khi báo bằng error — nó báo bằng wait_event_type = 'Lock' ở pg_stat_activity và bằng latency tăng từ phía application. Khi một incident xảy ra ("API treo, không ai biết tại sao"), thứ team cần trong 60 giây đầu là một bức tranh: PID nào đang đợi, đợi lock loại gì trên object nào, bị block bởi PID nào, PID block đó đang chạy query gì và đã giữ transaction bao lâu . pg_stat_activity một mình chỉ trả lời được nửa câu hỏi ("ai đang đợi"); pg_locks một mình chỉ trả lời nửa còn lại ("ai giữ gì"). Phải join hai view này — và bám theo pg_blocking_pids() — để dựng được blocking tree. Không có dashboard cho luồng dữ liệu này là lý do điển hình một production freeze kéo dài 30 phút thay vì 3 phút: incident commander phải mò ad-hoc bằng psql , gõ sai query, miss idle in transaction đang giữ AccessExclusiveLock của một migration nửa đời trước. Cơ chế hoạt động pg_locks là một view phơi nội dung trực tiếp của shared lock manager trong shared memory. Mỗi dòng là một lock request (đã granted hoặc đang chờ) thuộc một backend. Theo Postgres docs phần "System Views → pg_locks", các column then chốt: locktype ( relation , transactionid , tuple , virtualxid , advisory ...), relation (OID — join pg_class ), transactionid , virtualtransaction , pid (backend PID), mode ( AccessShareLock , RowExclusiveLock , ShareUpdateExclusiveLock , AccessExclusiveLock ...), granted (bool), fastpath (lock đi qua fast-path tránh shared lock manager), và waitstart (timestamp bắt đầu chờ — bổ sung sau v14, hữu ích để đo lock wait time mà không cần snapshot diff). pg_stat_activity là view phơi trạng thái runtime của mỗi backend: pid , usename , datname , application_name , client_addr , backend_start , xact_start , query_start , state ( active , idle , idle in transaction , idle in transaction (aborted) ), wait_event_type , wait_event , backend_xid , backend_xmin , qu

Most of the "AI for observability" work I see right now hands the language model the judgment. I think that's backwards. Feed it the alert, feed it some metrics, ask it what's wrong, what should be done, and let it make the judgement call. Based on my experience working with language models, I decided that inverting the process provides better results. The short version: in my alerting pipeline, the set of allowable classifications is fixed in deterministic Python, and the model has to pick from it. The LLM's only job is to turn a structured verdict into an easily digestible sentence. It never decides whether something is bad, how bad it is, or what category of problem it is. It narrates within a decision space the code has already locked down. TL;DR: Instead of letting an LLM decide what's wrong with an alert, I let deterministic Python make every operational decision and restrict the model to explaining the result in plain English. The code classifies, validates, and aggregates; the LLM only narrates. That keeps the data consistent, prevents hallucinated classifications, and ensures the monitoring pipeline continues working even if the model fails. The problem I run a small managed monitoring service. Alertmanager fires, a webhook lands, and historically that webhook produced a line like HighMemoryUsage on host web-vm, severity warning, which is accurate, but not terribly helpful. The person reading it still has to know what HighMemoryUsage implies, whether this host always runs hot, and whether to care. I wanted plain-English context attached to the alerts without altering the alert delivery process. The obvious move was to throw the whole alert at an LLM and ask it to explain. I tried that in the first iteration of this experiment, expecting it to be somewhat accurate, but not entirely reliable, and it did not disappoint, the model was confidently inconsistent. The same alert, fired three times, produced three different "root cause" categories. One run called a

Last month our visual-diff monitor flagged 47 changes on a client's homepage in one run. Forty-six of them were a rotating testimonial carousel that happened to land on a different slide each time the page was captured. One was real. If you've built or used any screenshot-based monitoring, you already know this problem. Two screenshots of the exact same, unchanged page rarely match pixel-for-pixel. Carousels rotate. Cookie banners fade in on a timer. Lazy-loaded images pop in a beat late. Ads shift half a pixel. Fonts render with slightly different anti-aliasing depending on what else the browser was doing. Diff two raw captures and you get a wall of "changes," and within a week nobody on the team opens the alert anymore. Why the obvious fixes don't work The first instinct is usually to loosen the pixel-diff threshold. That just trades false positives for false negatives - now a genuinely moved button or a broken layout has to clear the same bar as carousel noise, so you miss the thing you built the tool to catch in the first place. The second instinct is manual exclusion zones: tell the tool to ignore the carousel <div> , the ad slot, the cookie banner. This works until the page changes - a redesign moves the carousel, a new banner ships with a different selector, and you're back to noisy alerts plus a pile of dead config nobody remembers writing. The third "fix" is tolerating the noise, which is what most teams actually do in practice, and it's a big part of why visual regression tooling has a reputation for being more trouble than it's worth. Make the page prove it's stable before you trust anything about it The fix that actually moved the needle for us wasn't a smarter diff algorithm. It was refusing to treat a single screenshot as ground truth at all. Before any comparison happens, the page goes through a stabilization pass: known cookie/consent overlays get removed (we track a couple hundred variants at this point — cookie banner vendors are not standardized),

You wire up the OpenAI SDK, ship the feature, and it works. Three weeks later someone in finance forwards a screenshot of a bill that tripled and asks what happened. You open the provider dashboard, see one big number, and… that's it. No per-feature breakdown, no idea which change caused it, no way to tell whether it's a bug or just growth. I've watched this happen at enough teams that I now treat "we can't explain our AI bill" as a predictable stage every company hits about two months after their first LLM feature ships. Here's how to get ahead of it — starting with plain code, then the tradeoffs, then where a dedicated tool actually earns its keep. Disclosure up front: I work on StackSpend, which does the full version of this. I've kept the first 80% of this post vendor-neutral because most of it you can and should build yourself before you buy anything. The core problem: the bill is a single number, your costs are not Provider dashboards give you total spend over time. What you actually need to make decisions is spend broken down by the dimensions you care about: Per feature — is it the summarizer or the chat assistant that's expensive? Per customer / tenant — which accounts cost more to serve than they pay? Per model — how much are you spending on GPT-4-class vs cheaper models? Per environment — is a runaway staging job quietly burning money? None of those dimensions exist in the raw bill. You have to attach them yourself, at call time, because after the request is gone the context is gone with it. Step 1: capture usage at the call site Every major provider returns token usage in the response. The trick is to log it with your own business context attached — the feature name, the tenant, the environment. Here's the pattern in TypeScript with the OpenAI SDK: import OpenAI from " openai " ; const openai = new OpenAI (); // Prices per 1M tokens — keep these in config, they change often. const PRICING : Record < string , { input : number ; output : number } > = { " g

The Cloud Native Computing Foundation (CNCF) has announced the graduation of OpenTelemetry, elevating the project to the foundation's highest level of maturity and formally recognizing it as production-ready for enterprise use. By Craig Risi