标签：#monitoring

Adapted from my original Japanese article , with AI-assisted translation and editing. My manager: “Test this device.” (Doesn't really know the product or the technology.) Me: “Sure.” (Also doesn't really know the product or the technology.) If you've worked in infrastructure, that may sound familiar. I'm Goda, a network engineer sharing things I learned while figuring out the job. SNMP comes up in a lot of network device testing. For a while, my idea of an SNMP test was simple: Run snmpwalk . Watch a pile of OIDs and values scroll past. Mark SNMP as working. A screen full of output is reassuring. It certainly looks like something is being monitored. Then I was asked to write a test plan for a device. I added an item along the lines of “Confirm that information can be retrieved using SNMP” and sent it for review. The feedback was: Which OIDs will you use for CPU and memory? The customer will probably ask. You should at least cover those. That was when it clicked: a successful walk and successful retrieval of the metrics we need are two different things. My other thought was, “Fine, you write the test plan, then.” But the feedback was fair. Getting something back is not the same as getting what you need. What did the successful walk actually prove? snmpwalk is useful. Net-SNMP's tool uses GETNEXT requests to walk through a subtree starting from a specified OID. Net-SNMP manual If it successfully returns values, you've established that you could read those values under those test conditions . That matters. It does not, by itself, establish that your CPU and memory monitoring requirements are satisfied, or that every required OID is available. I had been treating a successful command as a much broader result than it actually was. Break “SNMP testing” into specific checks Today, I'd separate at least these questions: Question What to check Can I communicate over SNMP? Whether the device responds to the intended request under the defined conditions Can I monitor CPU? The

A process supervisor can tell you that a process exists. It cannot, by itself, tell you that a Minecraft player can join. I work on ChunkCraft, a Minecraft hosting project. Here is a small state model that helps keep operational status separate from player-facing guidance. Separate three questions Is the process alive? The container or service manager owns this signal. Has the game finished starting? Startup logs or a game-level probe provide this evidence. Can this player join? Client version, edition, whitelist and network reachability still matter. A useful state model is stopped → starting → ready , with failure and unknown states represented explicitly. Avoid converting a failed probe into “stopped”: a timeout means the observation failed, not necessarily that the server died. Tie each state to a next action Observed state Useful guidance Starting Wait for world loading; show recent startup progress Ready Show the complete connection address and expected version Unreachable or unknown Show when the last successful observation happened and offer diagnostics Player rejected Read the actual join error; check version and whitelist The same principle applies to control buttons. A copy-address action is helpful when the address exists and startup has completed. Showing it as the only instruction during startup invites repeated failed joins. Do not confuse observation with proof Even a successful game-level probe does not prove every player can reach the server. Likewise, a positive player-count sample proves someone was connected at that sample time; it does not identify that person or establish uninterrupted availability. Store observation timestamps alongside values. When a collector fails, preserve historical observations but mark them stale. A freshly rendered dashboard is not evidence of fresh underlying data. A small review checklist Does every status describe an observation we actually have? Is an unknown state distinguishable from a confirmed failure? Does th

Disk full alerts at 2am? Learn the exact commands to find what's eating your Linux server's disk space and fix it fast. You get the alert: disk usage at 94%. Your app starts throwing errors, logs stop writing, and databases refuse to accept new rows. Finding the culprit fast matters — but on a server with millions of files, knowing where to look is half the battle. Here's a systematic approach to track down disk hogs in minutes, not hours. Start With the Big Picture: df Before you dig into directories, confirm which filesystem is actually full. Run: df -h — shows all mounted filesystems with human-readable sizes df -h / — focus on the root filesystem df -i — check inode usage (a filesystem can be 'full' even with free space if inodes are exhausted) Pay attention to the 'Use%' column. If you see 100% on /var or /home but not /, that tells you exactly which mount point to investigate. Inode exhaustion — df -i showing 100% — is easy to miss and causes the same symptoms as a full disk, so always check both. Drill Down With du Once you know which mount point is full, use du to find the largest directories. Start from the top of that mount point and work down: du -sh /* 2>/dev/null — sizes of every top-level directory, errors suppressed du -sh /var/* 2>/dev/null — drill into /var if that's the culprit du -ah /var | sort -rh | head -20 — list the 20 largest files and folders inside /var The pattern is always the same: run du -sh on the suspicious directory, find the largest subdirectory, repeat one level deeper. You'll usually hit the real culprit within three or four iterations. Common offenders are /var/log (runaway logs), /var/lib/docker (unused images and volumes), and /tmp (applications that don't clean up after themselves). Find Large Files Directly With find Sometimes a single enormous file is the problem — a core dump, a forgotten database export, or a log that rotated incorrectly. Use find to surface files above a size threshold: find / -xdev -size +500M -ls 2>/de

How to Handle Anti-Bot Measures When Taking Screenshots Programmatically You send a request. The page loads. The screenshot comes back blank, or shows a CAPTCHA, or captures a "Please verify you're human" wall. This is one of the most common problems when building any screenshot pipeline. Here's what's actually happening and how to deal with it. Why headless browsers get flagged Bot detection works by looking for patterns that differ from real users. Headless Chrome has several tells: navigator.webdriver returns true by default Missing Chrome-specific properties like window.chrome Inconsistent screen dimensions (no monitor attached means no GPU info) Mouse events fire at pixel-perfect coordinates with no jitter Font fingerprints differ from headed browsers Modern detection services (Cloudflare, Akamai, Datadome) look for combinations of these signals, not individual flags. Spoofing one without the others often makes the fingerprint more suspicious, not less. The practical spectrum of detection Most sites fall into one of three categories: No active detection — a basic bot check via User-Agent string at most. Simple fix: set a realistic UA. Passive fingerprinting — loads a detection script, collects signals, blocks on second or third visit. You'll see this on news sites, e-commerce, media platforms. Active challenges — Cloudflare Turnstile, hCaptcha, reCAPTCHA v3 score-based. These require real interaction or a solving service. Know which category your target falls into before spending time on it. Fixes that work for most cases 1. Use a stealth plugin For Playwright, playwright-extra with puppeteer-extra-plugin-stealth patches the most common fingerprinting vectors: npm install playwright-extra puppeteer-extra-plugin-stealth import { chromium } from 'playwright-extra'; import StealthPlugin from 'puppeteer-extra-plugin-stealth'; chromium.use(StealthPlugin()); const browser = await chromium.launch(); This handles navigator.webdriver , window.chrome , and several other

In meinem ersten größeren Projekt waren wir stolz auf unsere Alarme. Wir hatten Hunderte davon. Jede Metrik, die uns eingefallen ist, bekam einen Schwellwert und eine Benachrichtigung. Wir dachten, das sei gründlich. In Wahrheit hatten wir uns eine Umgebung gebaut, in der niemand mehr hinschaute. Der Effekt kam schleichend. Erst überflog man die Alarme noch. Dann klickte man sie weg. Am Ende gab es einen eigenen Ordner, in den die ganzen Mails wanderten, ungelesen. Als dann ein echter Ausfall kam, ging seine Meldung im Rauschen unter, genau wie alle anderen. Wir hatten Alarme, aber wir hatten keine Alarmierung mehr. Das ist ein Unterschied. Seitdem behandle ich Alarme mit einer fast strengen Sparsamkeit. Meine Leitfrage lautet: Wenn dieser Alarm nachts losgeht, würde ich wollen, dass jemand aufsteht? Wenn die Antwort nein ist, dann ist es kein Alarm. Dann ist es höchstens ein Eintrag in einem Dashboard, das man morgens anschaut. Diese Trennung zwischen „jemand muss jetzt handeln" und „das ist zur Kenntnis" hat unsere Bereitschaft gerettet. Der zweite Grundsatz ist, auf Symptome zu alarmieren, nicht auf Ursachen. Nutzer merken nicht, dass eine CPU bei achtzig Prozent liegt. Sie merken, dass die Seite langsam ist oder Fehler zurückkommt. Also alarmiere ich auf das, was der Nutzer spürt: Latenz, Fehlerrate, gescheiterte Anfragen. Die hohe CPU ist dann eine Information für die Diagnose, kein Grund, jemanden zu wecken. Oft löst sich eine hohe Auslastung von selbst, ohne dass ein Mensch je etwas tun müsste. Am meisten hat mir geholfen, jeden Alarm mit einer ehrlichen Verfallsprüfung zu versehen. Alle paar Wochen gehe ich die Liste durch und frage bei jedem einzelnen: Ist der schon mal losgegangen? Hat daraufhin jemand etwas getan? Wenn ein Alarm seit Monaten nur Lärm macht und nie zu einer Handlung geführt hat, dann löse ich ihn auf. Das fühlt sich mutig an, ist aber das Gegenteil. Ein Alarm, dem niemand traut, schützt niemanden. Gutes Monitoring erkennt man nicht an der

A practical validation view for metric intervals, dimensions, notifications, suppression, and operator action An OCI alarm can be created correctly and still fail as an operating control. The issue may not be the alarm object itself. It may be the metric interval, a wrong dimension, an unconfirmed notification subscription, an absence query that fires for the wrong reason, or a message that reaches the team without telling them what action to take. This article is written from an OCI readiness and validation point of view. It is not a step-by-step Console walkthrough. The aim is to identify checks that should be confirmed before an alarm is treated as Production-ready. 1. The metric interval does not match the metric emission frequency This is one of the easiest alarm issues to miss. OCI documents a troubleshooting case where an alarm fires and then clears at the next interval even though the last data point exceeded the threshold. The cause is that the alarm interval is shorter than the metric's emission frequency. For example, if a metric is emitted hourly, an alarm interval that evaluates every minute may not behave the way the owner expects. The readiness check is to confirm the metric's documented emission frequency and set the alarm interval at the same frequency or longer when needed. Validate: Metric namespace and metric name Emission frequency for the selected metric Alarm interval in the MQL query Statistic used, such as mean, max, sum, or percentile Expected FIRING and OK behavior during testing Evidence can come from a metric-documentation or chart review, the alarm query, and a test record showing expected state behavior. The question for the owner is direct: does the alarm evaluate data at a rhythm that the metric can actually provide? 2. Dimensions include the wrong resource, or exclude the right one A metric query can be technically valid and still point at the wrong scope. In OCI Monitoring, dimensions help narrow metric streams. That is useful, but

A ticket calendar looks like an easy automation target: request a page, search for a date, and send an email when it appears. That implementation works until the first queue, partial response, stale cache or provider outage. Then it can quietly turn "I do not know" into "sold out" — or generate a false alert. I learned this while building MachuPing , an independent monitor for official Machu Picchu ticket availability. I am the maker. It does not sell, hold, reserve or buy admission; the official booking platform remains the source of truth. The useful abstraction is a small state machine: UNKNOWN -> CONFIRMED_UNAVAILABLE -> RETURNED_AVAILABLE ^ | | | v v +------------- PROVIDER_ERROR ------ ALERTED The exact labels will vary, but three rules matter. 1. Unknown is not unavailable Queues, timeouts, malformed payloads and incomplete calendars are observations about the monitor, not evidence about inventory. Persist them separately. A provider error should never close a date or trigger a reassuring "still sold out" message. 2. Match the user's real constraint "Machu Picchu is available" is too broad to be useful. Inventory is split by route, date, entry time and capacity. A valid transition requires a match for the selected combination, including the requested party size. This also prevents a common analytics mistake: counting every polling response or every seat-like value as a unique ticket. A state change is a state change, not proof of inventory volume. 3. Alert on a confirmed transition, not a snapshot The valuable event is not simply available . It is a move from a previously confirmed unavailable state to confirmed available. Persist an idempotency key for that combination so retries do not create duplicate email. Before sending, revalidate the observation when the provider permits it. The alert should still state the limitation plainly: availability can disappear before the traveller reaches official checkout. A practical event record At a module boundary, I pr

If you have ever built a health check, you have probably written something close to this: const res = await fetch ( url , { method : ' GET ' , signal : AbortSignal . timeout ( 10000 ) }); const isUp = res . status === 200 ; I ran a version of that for a while. It is wrong in at least five ways, and every one of them bit me while building an outage tracker for Indian services. This is a write-up of what actually breaks, because most monitoring tutorials stop at the snippet above. 1. The server answers, the service is dead The single biggest gap. 200 OK tells you a server returned a response. It tells you nothing about whether the thing a user came to do still works. A bank homepage can render in 400ms while UPI payments from that same bank are failing at the switch. Different systems, different teams, different failure modes. Your check is green and the feature is on fire. You cannot fully solve this from outside. What you can do is stop treating a 200 as proof of health, and stop displaying it as one. 2. 403 is not down Plenty of sites block automated requests deliberately. Bot protection, WAF rules, rate limits, geo rules. In India this is common on high-value government and travel portals. IRCTC is the obvious example. A naive checker marks these down permanently. Users learn to ignore your tool inside a week. 403 means the server is alive and refusing your specific request. That is different information from 500 , and treating them the same throws away the distinction that matters most: Code Server state What it tells a user 200 Alive, responded Little. The feature may still be broken. 401 / 403 Alive, refusing this request Usually nothing about the outage. Often your check being blocked. 404 Alive The path is wrong, not the service 429 Alive, rate limiting you You are the problem, back off 500 / 502 / 503 Broken, overloaded, or in maintenance Genuine signal 504 Something upstream did not answer Genuine signal, usually a dependency Timeout / DNS failure Unknown A

A sensor on my machine returned nothing at all — empty stdout, empty stderr, exit code 2 — on every invocation for 36 days. It was not crashed. It was not misconfigured. It was doing exactly what one line of well-intentioned code told it to do: announce a condition once . The line looked like this, and I suspect you have written it: if [ ! -f " $OFFLINEFILE " ] ; then echo "body context n/a — phone unreachable" > &2 touch " $OFFLINEFILE " fi exit 2 Read it as a rate limiter and it is obviously fine: don't spam the log with the same message every five minutes. Read it as what it actually is and it is a bug, because the guard does not limit a rate. It changes the tense of the sentence. Every number, code listing, and command output below was re-measured on the machine while writing this, not quoted from the commit that fixed it. Two of the things I expected to find turned out to be false; both are in section 6, and one of them is the most interesting part. 1. Present tense, past tense phone unreachable is a claim in the present tense . It is a statement about the world right now, and it is what a reader of this tool wants: is the body sensor readable at this moment? Wrapping it in [ ! -f "$SENTINEL" ] silently rewrites it into the past tense : the phone became unreachable, at some earlier point, at least once. That is a different proposition. It is true exactly once per transition and false forever after, which is why the guard can never fire twice, and why the sentinel's own mtime is the only surviving record of when the sentence was last true. The two propositions coincide on the first run. That is the whole trap. A first-time-only notice is indistinguishable from a live one for the length of one invocation, which is exactly the length of the test you will write for it. 2. What the reader got instead Here is the tool, before the fix, run twice in a row against a phone that is genuinely away. I pulled the pre-fix version straight out of git into a scratch path and ra

A Windows 11 VM in Proxmox VE was showing nearly 100% memory usage in monitoring. Inside Windows Task Manager, however, actual memory usage was only around 30–50% . At first glance, that looks like a monitoring problem. It wasn't. The issue was in the VM configuration: the PVE Ballooning Device had been disabled , which meant Proxmox VE was not receiving the guest memory statistics needed to reflect the actual Windows memory state. I encountered this while monitoring a Proxmox VE environment with OpsHome NOC. This post documents how I traced the discrepancy and fixed it. The symptom On the same Proxmox VE host, the memory usage of Ubuntu VMs looked normal. One Windows 11 VM was different. The VM had 24 GB of RAM configured, but the monitoring result remained close to: Memory: 100% Used: about 24.2 GB Total: 24 GB Inside Windows 11 Task Manager, however, the VM was clearly not using all of its memory. The difference looked roughly like this: Monitoring: 90%–100% Windows 11: 30%–50% That is too large a difference to treat as a normal sampling variation. If you encounter something similar, especially when Linux VMs on the same Proxmox host look normal, do not immediately assume: Windows has a memory leak The monitoring threshold is wrong The monitoring application is calculating memory incorrectly The more important question is: Is Proxmox VE actually receiving the correct memory statistics from the Windows guest? Checking BalloonService inside Windows 11 For Proxmox VE to obtain useful guest memory statistics from a Windows VM, the VirtIO Balloon driver and its related Windows service need to be available. Inside Windows 11, I opened PowerShell and checked BalloonService: Get-Service * balloon * The result showed: Running BalloonService So the Windows-side BalloonService was already installed and running. At this point, the guest-side service did not appear to be the problem. The next step was to check the VM configuration on the Proxmox side. Checking the Proxmox VE

Linux server running out of memory? Learn how to diagnose and fix high memory usage with real commands — before it takes down your app. Your app starts slowing down, the OOM killer fires, or your monitoring page turns red — and the culprit is memory. High memory usage on a Linux server is one of the most common production crises for small teams, and it's easy to misread. Linux intentionally uses most of your RAM for caching, so a server showing 95% memory used isn't necessarily in trouble. But one that's exhausting real working memory and swapping is. Here's how to tell the difference and actually fix it. Step 1: Get a Clear Picture of What's Using Memory Start with the basics. Run 'free -h' to see total, used, free, and available memory. Focus on the 'available' column — that's the real number. It accounts for reclaimable cache and is far more useful than 'free'. free -h — quick overview of RAM and swap usage vmstat 1 5 — five one-second snapshots; watch the 'si' and 'so' columns for swap-in and swap-out activity cat /proc/meminfo — full breakdown including Slab, PageTables, and AnonPages If swap is actively being used (si/so values above zero consistently), your server is genuinely memory-constrained. That's different from swap space existing but sitting idle. Step 2: Find the Processes Eating Your RAM Once you know memory is tight, you need to know what's consuming it. Run 'ps aux --sort=-%mem | head -20' to list the top 20 processes by memory percentage. For more detail on actual RSS (resident set size) in human-readable form: ps -eo pid,ppid,cmd,%mem,rss --sort=-%mem | head -20 RSS is the memory a process actually holds in RAM — not virtual memory, which is often misleadingly large. Another useful tool is 'smem', which calculates PSS (proportional set size) and gives a fairer view when processes share memory libraries. Install it with 'apt install smem' or 'yum install smem', then run 'smem -r -k | head -20'. Look for processes with unexpectedly high RSS. A Nod

Seri: Proxmox VE Cluster ve Corosync | Hafta 5 Serinin adı "Cluster ve Corosync"; ama dört modüldür ağırlık HA Manager, resource affinity ve CRS'teydi, Corosync'in kendisine (redundant link'ler, izleme araçları) hiç dönmemiştim. Bu modülde iki konuyu birleştirip derinlemesine işledim: birden fazla corosync link'i tanımlayıp gerçekten birini kesip diğerinin devralmasını kanıtlamak, ve günlük operasyonda kullanılacak izleme araçlarını tek tek denemek. İkisi de planladığımdan çok daha fazla soru açtı; biri yanlış bir config anahtarı yüzünden saatler süren bir araştırmaya dönüştü, diğeri ise hiç beklemediğim bir kilitlenme keşfiyle bitti. Bölüm 1: Redundant Corosync Links Kurulum: İkinci Link'i Eklemek Şu ana kadar cluster'ımızda tek bir corosync link'i vardı ( link1 , izole corosync-net ağı). Management ağını ( 192.168.122.x ) link0 olarak ekleyip gerçek bir yedeklilik kurdum; /etc/pve/corosync.conf 'u kopyalayıp düzenleyip atomik olarak yerine taşıdım: cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new # nodelist'teki her node'a ring0_addr ekledim, totem'e ikinci bir interface bloğu ekledim mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf Doğrulama: corosync-cfgtool -s LINK ID 0 udp addr = 192.168.122.11 status: ... connected ... connected LINK ID 1 udp addr = 10.10.10.11 status: ... connected ... connected Teknik olarak başarılı; iki link de bağlı. Ama log'a dikkatlice bakınca, mimarimizin niyetini tersine çeviren bir şey oldu: [KNET ] rx: host: 3 link: 0 is up [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) link_mode: passive modunda, öncelik eşitken düşük numaralı link kazanıyor . link0 'ı sonradan eklediğim için, o Corosync'in asıl trafiğini üstlenmiş; Modül 0'da özellikle izole ettiğimiz corosync-net ( link1 ) sessizce yedek konuma düşmüştü. Yanlış Anahtar, Saatler Süren Bir Araştırma Bunu düzeltmek için link1 'e daha yüksek öncelik vermeye çalıştım: interface { linknumber : 0 priority : 5 } interface { linknumber : 1 priority : 10 } İşe yaramadı. cor

Most monitoring watches for something bad to appear: a 500, a timeout, an expired certificate, a slow response. A heartbeat monitor does the opposite. It watches for something good to stop appearing . Your cron runs, your backup completes, your embedded device phones home, your queue worker drains — and each of those pings a URL to say "I'm still alive." The monitor's job is to notice when the pings go quiet. That inversion is the entire value. A cron that fails throws an error you can catch. A cron that stops being scheduled — the box got reimaged, the systemd timer got disabled, the container never came back after a deploy, the account got suspended for an unrelated billing issue — throws nothing at all. There is no log line, no exception, no non-zero exit. There is only the absence of the thing that used to happen. You cannot alert on an event that does not fire. You can only alert on the silence. So heartbeat monitoring looks trivial: store a timestamp on every ping, and if now - last_seen > expected_interval , fire an alert. It is about ten lines. And it is exactly those ten lines that will let 43 days of downtime pass without a second word — because the hard part of a dead-man's switch is not detecting the death. It is staying loud after it. I know because it happened to our own. Three states, and why the third one must stay silent Start with the check itself. A naive heartbeat has two states — alive or dead — and both are wrong at the edges. The real answer set has three: alive — a beat arrived within period + grace . Everything is fine. dead — the last beat is older than period + grace . The thing stopped. Page someone. unknown — the monitor exists but has never received a single beat. That third state is where two-state heartbeat monitors self-immolate. A brand-new heartbeat you just created has no last_seen timestamp. If your rule is "alert when last_seen is too old," a null last_seen is infinitely old, so the monitor pages you the instant you create it —

Last week I turned my data audit into a build step : a check that runs before anything else and fails the build when the database and any static copy of my travel site's legal-status data disagree. It ended the era of the site contradicting itself. It did nothing about the site agreeing with itself on something false. That's not a hypothetical. The most expensive error the whole project found was a country whose law changed in January while every copy on my site — database, data files, search index — kept saying the old thing in perfect unison. Internal consistency was the camouflage . No diff between my own sources could ever have caught it, because every internal source was equally behind the world. A build gate proves agreement. Agreement is not truth. Something has to look outside. You can't diff against the world, but you can sample it The naive version of "look outside" is another audit — a human session checking primary sources jurisdiction by jurisdiction. I've done three of those now, and I know exactly what they're worth: they're correct the day they ship and they decay from that morning on. Laws don't change on my audit schedule. So the outside check became what the inside check became: a scheduled job. Once a week, a script asks a web-connected model — one that searches and cites, not one answering from training memory — for the current legal status of about fourteen jurisdictions, and compares each answer to the corresponding database row. Fourteen, not all 271, because the selection is doing the real work: A hot list is checked every single run: the highest-traffic pages plus the jurisdictions with active legislative motion — the places where being a month stale costs the most. Everything else sits on a rotating cursor : eight per run, round-robin, so every row on the site gets sampled roughly twice a year without any run costing more than a few cents. The whole thing runs on about seven cents a week. Two rules were non-negotiable, both inherited from

When every unauthorized request becomes the same dashboard line, diagnosis turns into guessing. ASP.NET Core 10 authentication metrics give me a better split: did the handler have nothing to authenticate, reject supplied credentials, or accept them? That distinction matters because a client deployment that drops credentials needs a different response from a surge of malformed or expired credentials. ASP.NET Core 10 added built-in authentication and authorization instruments to System.Diagnostics.Metrics . I can collect them without rewriting each handler, and I can lock their behavior into an offline test before wiring up a production exporter. Why one 401 hides two different problems A protected endpoint normally challenges an unauthenticated caller. The final status is 401 whether the caller sent nothing or the handler rejected what it received. The authentication duration histogram exposes the missing context through aspnetcore.authentication.result : Result What the handler reported A common interpretation none No authentication result No applicable credentials were available failure Authentication failed Supplied credentials were rejected or processing failed success A principal was created Authentication completed successfully _OTHER Another framework result Preserve it as an explicit catch-all none is a handler result, not a universal synonym for “missing Authorization header.” A policy scheme or custom handler can make a different choice. I verify the behavior of the schemes I actually deploy instead of building an alert from the label alone. Likewise, success means the handler produced an authentication ticket. Authorization can still deny that principal, so it does not promise a 2xx response. The separate aspnetcore.authentication.challenges counter answers another question: how often was a scheme challenged? Both a none result and a failure result can be followed by a challenge, so challenge count cannot replace the result split. A challenge is an authent

Sarvar's post about discovering a hidden retry in a 5-agent pipeline (one agent taking 22.6s while others took 5s) is a perfect case study in why observability infrastructure matters for agentic systems. Here's what jumped out: Agent-as-black-box is dangerous. When you string together multiple agents, you lose visibility into retry logic, backoff strategies, and cascade failures unless you instrument at the span level. The latency wasn't in the agent logic itself; it was in the retry envelope. Span hierarchy exposes the invisible. Sentry's approach of grouping spans hierarchically made the problem visible at a glance. Without it, you'd see "agent took 22.6s" and assume it was compute-bound. With hierarchy, the retry pattern was obvious. This scales badly across agents. In a 5-agent system, one bad retry strategy can block or cascade. Add error handling, timeout logic, and fallback chains, and you're building a retry forest no one fully understands. The observability debt compounds. The fix is cheap, the insight is priceless. Once Sarvar knew what was happening, tuning retry counts or backoff curves took minutes. The time cost was finding it. Takeaway: If you're building multi-agent systems, instrument early. Span-level observability isn't optional; it's the difference between "it's slow" and "here's why, and here's the fix."

Originally published on hexisteme notes . A status column in one of my agent fleet's ledgers said 30 items were queued to publish. A working session that day stated a backlog close to a month at the fleet's normal rate and deferred the work that keeps posts flowing into the queue. At that moment the ledger showed the same backlog. That exact numeric match suggests — but does not prove — that the ledger informed the decision. The real number of items actually waiting was 6. At one post published per day, that is six days of runway, against a low-water alarm configured to fire at 3. The gap came from a status value that was never advanced after publication, not from the queue-file count itself. A column just quietly stopped meaning what everyone assumed it meant, and by the time it mattered, it had been wrong for a while. The pipeline, briefly The fleet runs a small publishing pipeline: a draft gets written, a promotion step validates it and drops a file into a queue directory, and a scheduled job runs once a day, picks the oldest file in that directory, publishes it, moves the file into a published folder, and appends one line to a log. Alongside the queue directory sits a separate ledger: a flat TSV file, one row per item, with a status column meant to track where each item sits in its life — staged, queued, published. Two different things track the same concept: the files actually sitting in the queue directory, and a column in a table that is supposed to describe them. Where it broke Exactly one piece of code writes status=queued : the promotion step, at the moment an item enters the queue. Nothing else ever changes that value afterward. The daily publish job moves the file and writes to the log; it never opens the ledger. Nobody had assigned any code the job of setting the status forward to published . So queued stopped meaning "currently waiting." It came to mean "was queued at some point," which, once true, is true forever. Every item that had ever passed throu

Use a kill switch inside the checker when your uptime probes start amplifying an incident — one feature flag, read on every tick, that can disable the noisy checks and stop the retries at the source. Reach for tuned backoff and jitter instead when the retry storm stays inside a single process and never fans out onto a dependency somebody else is paging for. Both are cheap to build. Only one of them lets you quiet a polling client while its target is already on fire. I run cron and queue infrastructure, so most of my pages arrive as either "the job didn't run" or "the job ran four times." Health checking sits in the same family of problems: a small, frequent, automated request that multiplies badly when something upstream changes shape. What follows is the runbook I settled on after a fleet of pollers turned a non-incident into a real one — the failure mode, where the switch belongs, the implementation, and how to verify the flip before you walk away from the terminal. What actually turns a polling client's uptime checks into a retry storm? Amplification. A single check is one request every 15 or 30 seconds, which nobody notices; a fleet of checks with retries layered on top is a synchronized load generator pointed at whatever you decided was important enough to monitor. The math is unkind. Take 40 instances, a 5s interval, and 3 retries per failed attempt, and a dependency that normally handles a trickle of health traffic suddenly sees a couple thousand requests a minute — all of them arriving at the exact moment it's least able to absorb them. Retries stack on top of the polling interval rather than replacing it, and because every poller sees the same failure at the same time, they all back off together and return together. The Google SRE book calls out this shape under cascading failures, and the load pattern that comes out of it looks nothing like organic traffic: sawtooth spikes, perfectly aligned, growing until something sheds load. The worst one I've dealt wit

TL;DR: Grafana Agent reached End-of-Life on November 1, 2025 and has been replaced by Grafana Alloy. Alloy consolidates Agent's Static mode, Flow mode, and Kubernetes Operator into a single collector built on the OpenTelemetry Collector while maintaining native support for Prometheus and Loki. If you're using Flow mode, migration is relatively straightforward. If you're using Static mode, the migration process will involve reviewing and testing the converted configuration. Before switching over, verify relabeling rules, recheck resource usage, and confirm that Prometheus and Loki are receiving the same data and labels as before. If you're still running Promtail, it's worth migrating both to Alloy at the same time since Promtail is also End-of-Life. If you deployed Grafana Agent a couple of years ago, there's a good chance you haven't thought about it since. It quietly collects metrics, ships logs, and generally stays out of the way. What you may not realize is that Grafana Agent reached End-of-Life on November 1, 2025. That includes Static mode, Flow mode, and the Kubernetes Operator. Grafana Labs has stopped creating bug fixes, security patches, and official support. If you're still running it, your collection layer is probably still performing normally, but is now unsupported. That doesn't necessarily mean it will stop working tomorrow, plenty of unsupported software continues running for years. It does mean you're taking on the risk yourself, especially as the rest of your monitoring stack continues to evolve. This article covers why Grafana Labs replaced Agent with Alloy, what actually changes during the migration, and where people tend to run into problems. Why Grafana Agent was deprecated One of the biggest issues with Grafana Agent is that it was essentially three agents, not one product: Static mode, which used YAML and looked similar to Prometheus. Flow mode, which introduced a component-based configuration using River. The Kubernetes Operator, which manage

TL;DR If you're a five-person startup shipping a Node.js API and you want a metrics dashboard by Friday, send your telemetry to a managed backend and keep only the instrumentation layer inside your own repo. The alternative — standing up a time-series database, an object store for long-term blocks, and a dashboard service — puts three more components on an on-call rotation that hasn't earned its first SLO yet. Settle the wire format now and treat the backend as a config line you can change later. I own the platform team's roadmap, which in practice means I'm the person who defends the monitoring bill in a budget review and also the person who gets paged when a disk fills at 03:00. Those two jobs pull in opposite directions, and most of the advice online is written by people who only hold one of them. Usually the pager wins the argument. Should a startup run its own metrics stack, or pay for a managed dashboard? Start with capacity, because that's the step everyone skips before signing anything. A moderately instrumented Node.js API — say 40 HTTP routes, two queue workers, default runtime and event-loop metrics, one latency histogram with ten buckets — sits somewhere around 3,000 to 8,000 active series per process. Multiply by replicas. Multiply again by every environment you keep alive, including the staging cluster nobody admits to. You are at 50k active series before a single engineer has written a custom counter, and a self-hosted scraper will chew through that on a 2 GB VM without noticing. It will still be fine at 500k. Past a few million active series you're into sharding, remote storage, and a retention argument with whoever pays for object storage — that's the point where the self-hosted route stops being free and turns into a project with a headcount attached. None of that work is hard. It's just never zero. Dimension Self-hosted stack Managed metrics backend Time to first dashboard 1–3 days under an hour Who owns retention you, plus the storage bill vendor