标签：#sre

I run about thirty scheduled jobs on a single Windows box. Some are scrapers, some generate content, some are trading bots, some just check that the other jobs are alive. Most of them were written and are maintained by an AI coding agent that I let run unattended. Over three months, every one of the failures below reported success . The scheduler said LastTaskResult = 0 . The logs looked fine or didn't exist. And nothing had happened. If you only take one thing from this post: stop checking exit codes, start checking artifacts. I'll get to why at the end. First, the seven ways I got lied to. 1. The wrapper that always returns 0 To stop console windows flashing on my desktop every few minutes, I wrapped each scheduled task in a tiny VBScript launcher: Set WshShell = CreateObject ( "WScript.Shell" ) WshShell . Run "cmd /c "" python job.py >> job.log 2>&1 "" " , 0 , True 0 hides the window. True waits for completion. I assumed True also meant the exit code came back. It does not. WshShell.Run used as a statement discards the return value, so wscript.exe exits 0 no matter what the child did. I found this because a content pipeline had been dead for five days while the scheduler reported green every single day. The fix is to call Run as a function and pass the value out: Set WshShell = CreateObject ( "WScript.Shell" ) exitCode = WshShell . Run ( "cmd /c "" python job.py >> job.log 2>&1 "" " , 0 , True ) WScript . Quit ( exitCode ) Note the parentheses — required when you're taking a return value. After fixing this across 17 launchers, one task showed a non-zero result for the first time in its life . It had been failing for weeks. 2. The last line of your batch file overwrites the exit code Fixed the launcher, still got false greens. The next layer down was a .cmd shim: node pipeline .js >> run .log 2 >& 1 echo [ done ] exit code %errorlevel% >> run .log That echo is the last command, echo always succeeds, so the batch file returns its exit code — zero — regardless of wh

Short answer: The PDF processing concepts developers should understand before designing reliable image asset extraction under load are object-versus-render semantics, fidelity contracts, byte-and-pixel admission limits, bounded concurrency, deadlines, idempotency, and separate verification for assets and flattened pages. For a gaming backend, that distinction matters when a player-submitted PDF feeds both a filled, flattened form and a set of review thumbnails. The form output needs visual consistency; the asset workflow needs stable tail latency under a burst. One operation should not quietly inherit the other operation's resource profile. The operational recommendation is concrete: admit work by bytes and estimated pixels, cap concurrent rendering separately from object extraction, store content-addressed outputs, and make every stage restartable. Don't promise synchronous completion merely because a single small PDF finishes quickly on a laptop. The unit of work matters. What PDF processing concepts determine reliable image asset extraction latency under load? A PDF is better treated as an object graph than as a folder of pictures. A page can refer to reusable image objects; it can also contain inline image data, transparency masks, clipping instructions, transformations, and color information that changes how pixels finally appear. An extractor that only walks obvious page resources may return encoded assets without reconstructing the composition a viewer shows. Conversely, a renderer resolves the page's visual instructions but turns vector art and text into pixels, which may be unnecessary when the caller asked for original image assets. That gives the workflow two different contracts. Object extraction returns embedded image payloads plus enough metadata to identify dimensions, placement, masks, and page references. Page rendering returns the visible page at a declared resolution and color policy. Calling both results “image extraction” creates an SLO problem

The startup wants to do for IT infrastructure what Cursor did for software engineering.

A syndication job noticed before I did A scheduled task publishes one blog post a day to a developer community. It fetches the article from my own site, converts it, and posts it. At 10:00 it failed four times with this: Server error '521 <none>' for url 'https://neuragrowth.co/blog/schema-grammar-ceiling/' 521 is Cloudflare saying the origin server did not answer. So the interesting failure was not in the syndication job at all. My whole site was down, and had been for over three hours by then. The server itself was fine: four days of uptime, load under 0.2, disk at eight percent. But systemctl is-active nginx said failed , and nothing was listening on 80 or 443. nginx resolves your upstreams before it starts The journal had the whole thing in three lines: 06:49:54 systemd[1]: Stopping nginx.service... 06:49:54 nginx[36027]: [emerg] host not found in upstream "example-backend.tld" in /etc/nginx/sites-enabled/site:104 06:49:54 nginx[36027]: nginx: configuration file test failed Line 104 was a small proxy I had added months earlier so the public site could forward one form endpoint to a backend on a different host without revealing its name: location = /api/lead-capture { proxy_pass https://example-backend.tld/api/lead-capture ; proxy_ssl_server_name on ; proxy_set_header Host example-backend.tld ; } When proxy_pass contains a literal hostname, nginx resolves it while parsing the configuration , and treats failure as a fatal config error. That resolution happens inside ExecStartPre=/usr/sbin/nginx -t , so a name it cannot look up means the unit never starts. The config was not wrong. It was valid before the restart and valid after, and nginx -t passed by hand seven hours later. It was invalid for about one second. Why DNS was gone for exactly that instant Ten seconds of journal, reconstructed: 06:49:44 apt-daily-upgrade.service starts 06:49:53 "Reexecution requested ... (unit apt-daily-upgrade.service)" 06:49:53 systemd reexecuting (it had just upgraded itself) 06:49

Originally published on Medium It's 2:51am. The incident is resolved. Error rate is back to zero, the rollback worked, and your on-call pager has finally gone quiet. Now you have to write the postmortem. If you've been in SRE or DevOps for any length of time, you know this feeling. You're exhausted, your brain is running on adrenaline fumes, and somewhere in the back of your mind you know that what you write in the next hour is going to be read by engineers, product managers, and probably a VP or two. It needs to be clear, blameless, specific, and actionable. Most of us write it badly. Not because we're bad at our jobs — because we're human beings who just spent two hours firefighting and now we're staring at a blank document at 3am trying to remember the exact sequence of events. There's a better way. The Problem With How We Write Postmortems The standard postmortem template is a solved problem. Every company has one. Timeline, root cause, contributing factors, action items — we all know the structure. The hard part isn't the structure. It's the writing. Specifically: Reconstructing the timeline from a chaotic Slack thread where half the messages are noise Writing the root cause narrative in plain language when your brain is still in technical mode Generating action items that are actually specific and assignable instead of vague gestures toward improvement Translating all of it into an executive summary that a non-technical VP can understand without losing the technical accuracy Each of these is a hard writing task under normal circumstances. At 2am after an incident they're brutal. What Changed for Me I started treating postmortem writing like any other repetitive engineering task: I built a system for it. Specifically, I built a set of AI prompts designed for the exact scenarios SREs face. Not generic "write me a postmortem" prompts — structured prompts that work with the raw material you actually have in front of you at the end of an incident. The key insight w

In mission-critical infrastructure, latency isn't just a metric—it's the difference between a resolved incident and a catastrophic outage. Whether you are managing an SRE team handling cluster failures or a teleassistance platform routing domestic SOS alerts, the core engineering challenge remains identical: getting a human's attention in milliseconds without administrative friction. This technical breakdown explores how we architected a high-availability voice routing engine using Cloudflare Workers and Twilio, bridging the gap between hardware teleassistance and DevOps incident workflows. The Dual-Use Architecture: From Teleassistance to SRE Paging Our platform core serves two distinct but structurally identical needs: Senior Safe: A Chilean domestic teleassistance product where an SOS trigger must reach a family guardian instantly. DevOps On-Call: An infrastructure alert triggered via Grafana or UptimeRobot webhooks that must wake up an engineer at 3 a.m. The blast radius differs (a household vs. a production database), but the technical path is identical. To solve this at scale without charging steep "per-seat" licensing models that penalize growing squads, we built the entire pipeline on serverless isolates. Bypassing Cold Starts with Edge Ingest When an emergency happens, you cannot afford to wait for a virtual machine or container to boot. The public ingest pipeline lives directly on Cloudflare Workers ( api.wakeupdev.com ). Because V8 isolates are kept warm globally across the edge network, there is zero Lambda-style cold start penalty on the first page. The ingest contract is minimal: Authentication: Handled via an x-api-key header. Payload: Raw text or JSON (capped at 4,000 characters). Execution: Credits are consumed atomically in a global Postgres layer before the voice cascade is scheduled. An HTTP 202 Accepted status code guarantees that the credit is validated and the call flow is in flight. Solving the Voicemail Problem: True Human Acknowledgement A

I wrote this for the All Things Agentic Hackathon. Every incident-response demo you have seen ends the same way: something breaks, the agent fixes it, everyone applauds. I want to show you the opposite. Here is my agent, at 95% confidence, having correctly diagnosed a bad deployment, deciding not to roll it back. That refusal is the whole project. The question underneath At 3am an alert fires. An engineer wakes up, reads several hundred log lines, correlates them against recent deploys, and rolls something back. Most of it is mechanical. It is an obvious target for automation. But "automate it with an LLM" does not dissolve the problem, it relocates it. The new question is: how much would you let an agent change in production without asking you first? Give it too little and it is a chatbot that writes summaries. Give it too much and one confidently wrong diagnosis takes down your service at 3am with nobody watching. I named the project Sonjomon — Bengali for restraint. The autonomy ladder An agent should not have one blanket permission level. How far it may act alone is a function of two things: how confident it is, and how much damage the proposed action does if that confidence turns out to be wrong. tier = f(confidence, blast_radius) OBSERVE record findings, take no action SUGGEST recommend to a human, do not execute APPROVE stage the action, execute on explicit approval ACT execute now, then verify independently A restart is medium risk — reversible in seconds. A rollback is high risk — it shifts production traffic, and a needless rollback during a real outage extends it. Deleting data is critical, and no confidence level unlocks it. Six conditions can only ever push the tier down, never up: the blast-radius ceiling, thin evidence, a similar action that just failed, a third attempt at the same fix, a stale incident, and a global dry-run switch. Nothing pushes it up. A wrong action is far more expensive than a missed one. Three things the model does not control It

📝 Originally published (in Japanese) at forge.workstyle.tech . Building an Unmanned System for 3D Avatar Live Streaming We're developing an unmanned system where 3D avatars automatically handle live streaming. The system boots up a cloud GPU pod at the scheduled start time, the renderer assembles and streams the video, and then the pod is discarded when the segment ends. Since there's no human oversight, three factors directly impact the success of the business and service quality: "how many avatars can run simultaneously," "how the system recovers from failures," and "how quickly it starts up." These questions couldn't be answered through estimates alone. Renting a GPU for a few hours costs only a few hundred yen. In this article, we'll share three stories of how we measured and designed the system, following the structure of "stumbling block → cause → solution." Capacity : How many avatars can run on a single GPU? The answer is 4, at a monthly cost of ¥7,600 per avatar. However, the bottleneck wasn't the GPU. Reliability : Despite using the same image, some hosts crashed every 60 seconds. We implemented a mechanism to automatically switch to a different host. Startup Speed : Reduced the time from pod startup to streaming start from 4 minutes to 95 seconds. These three aspects seem independent but are actually interconnected. Faster startup enabled practical host switching, and understanding capacity allowed us to set prices. Let's dive into each one. 1. How Many Avatars Can Run on a Single GPU? - Measured Result: 4 The first number we desperately needed was "how many avatars can run on a single GPU." Without this, we couldn't determine pricing, and without pricing, we couldn't assess the business viability. Estimates were useless, so we measured it. Here are the results: Item Measured Value GPU RTX 4000 Ada (Community type, $0.28/hour) Simultaneous Streams 4 avatars maintaining 720p30 in real-time (Recorded segment: 89 seconds / 89 seconds) GPU Usage 26% Bottlenec

A ticket calendar looks like an easy automation target: request a page, search for a date, and send an email when it appears. That implementation works until the first queue, partial response, stale cache or provider outage. Then it can quietly turn "I do not know" into "sold out" — or generate a false alert. I learned this while building MachuPing , an independent monitor for official Machu Picchu ticket availability. I am the maker. It does not sell, hold, reserve or buy admission; the official booking platform remains the source of truth. The useful abstraction is a small state machine: UNKNOWN -> CONFIRMED_UNAVAILABLE -> RETURNED_AVAILABLE ^ | | | v v +------------- PROVIDER_ERROR ------ ALERTED The exact labels will vary, but three rules matter. 1. Unknown is not unavailable Queues, timeouts, malformed payloads and incomplete calendars are observations about the monitor, not evidence about inventory. Persist them separately. A provider error should never close a date or trigger a reassuring "still sold out" message. 2. Match the user's real constraint "Machu Picchu is available" is too broad to be useful. Inventory is split by route, date, entry time and capacity. A valid transition requires a match for the selected combination, including the requested party size. This also prevents a common analytics mistake: counting every polling response or every seat-like value as a unique ticket. A state change is a state change, not proof of inventory volume. 3. Alert on a confirmed transition, not a snapshot The valuable event is not simply available . It is a move from a previously confirmed unavailable state to confirmed available. Persist an idempotency key for that combination so retries do not create duplicate email. Before sending, revalidate the observation when the provider permits it. The alert should still state the limitation plainly: availability can disappear before the traveller reaches official checkout. A practical event record At a module boundary, I pr

On ne gère pas ce qu'on ne mesure pas. C'est l'une des premières leçons de l'exploitation, et pourtant je l'ai apprise à l'envers, en pilotant à l'aveugle bien trop longtemps. Sans mesure, tu ne sais pas si un système va bien. Tu le supposes. Il tourne, personne ne se plaint, donc tout va bien — jusqu'au jour où quelque chose se dégrade lentement, sous le radar, et où tu ne l'apprends que lorsque c'est déjà une panne. La lente fuite de mémoire, le disque qui se remplit, la latence qui grimpe d'une milliseconde par semaine : rien de tout cela ne crie. Ça glisse. La mesure transforme les suppositions en faits. Un tableau de bord, quelques alertes bien choisies, et soudain tu vois le problème arriver au lieu de le subir. Tu n'attends plus que l'utilisateur t'apprenne que ton système est cassé ; tu le sais avant lui. Mais il y a un piège que j'ai appris à éviter : mesurer trop. Cent métriques que personne ne regarde ne valent pas mieux que zéro. Le bruit noie le signal, et les alertes qui se déclenchent sans raison finissent par être ignorées — jusqu'à celle qui comptait vraiment. Bien mesurer, ce n'est pas tout mesurer. C'est choisir les quelques signaux qui prédisent réellement un problème. Alors, avant de bâtir la prochaine chose, demande-toi comment tu sauras si elle va mal. Si la réponse est « quelqu'un finira par le remarquer », tu ne la gères pas encore. Tu espères. Et l'espoir n'est pas une stratégie d'exploitation. – Serguey Shinder

A sensor on my machine returned nothing at all — empty stdout, empty stderr, exit code 2 — on every invocation for 36 days. It was not crashed. It was not misconfigured. It was doing exactly what one line of well-intentioned code told it to do: announce a condition once . The line looked like this, and I suspect you have written it: if [ ! -f " $OFFLINEFILE " ] ; then echo "body context n/a — phone unreachable" > &2 touch " $OFFLINEFILE " fi exit 2 Read it as a rate limiter and it is obviously fine: don't spam the log with the same message every five minutes. Read it as what it actually is and it is a bug, because the guard does not limit a rate. It changes the tense of the sentence. Every number, code listing, and command output below was re-measured on the machine while writing this, not quoted from the commit that fixed it. Two of the things I expected to find turned out to be false; both are in section 6, and one of them is the most interesting part. 1. Present tense, past tense phone unreachable is a claim in the present tense . It is a statement about the world right now, and it is what a reader of this tool wants: is the body sensor readable at this moment? Wrapping it in [ ! -f "$SENTINEL" ] silently rewrites it into the past tense : the phone became unreachable, at some earlier point, at least once. That is a different proposition. It is true exactly once per transition and false forever after, which is why the guard can never fire twice, and why the sentinel's own mtime is the only surviving record of when the sentence was last true. The two propositions coincide on the first run. That is the whole trap. A first-time-only notice is indistinguishable from a live one for the length of one invocation, which is exactly the length of the test you will write for it. 2. What the reader got instead Here is the tool, before the fix, run twice in a row against a phone that is genuinely away. I pulled the pre-fix version straight out of git into a scratch path and ra

The page says that a property manager never received a welcome email. The useful signal should have arrived earlier, when that tenant's sending domain or delivery-event polling stopped matching the expected state. Short answer: keep welcome-email templates in the application when review history and portability matter most; use provider-owned templates when authorized non-engineers need to edit and preview copy, then select a transactional email provider that supports your chosen ownership model, per-domain management, and occasional batch sends. For a multi-tenant property SaaS, don't let the provider choose the template owner by accident. The reliable design is small: one authoritative template, one tenant-to-domain mapping, and one delivery ledger keyed by an application-generated message ID. Provider selection comes after those decisions. This ordering matters because a successful API request cannot prove that the correct branded message reached the correct property manager. Ownership comes first. How should multi-tenant SaaS welcome email templates be owned? Start with the people allowed to change the welcome message. Application-owned templates put markup, variables, tests, and review history beside the workflow that creates a manager account. They fit when a copy change must ship with a schema change, security-sensitive wording requires code review, or provider portability is a firm requirement. The catch is that a typo correction joins the engineering release path, and the team must build or adopt its own preview step. Provider-owned templates invert that arrangement. A lifecycle or support team can edit copy inside a controlled delivery workflow, and template preview lets a junior developer inspect the branded result before activation. Template identifiers and variable contracts then become deployed configuration. Rollback means selecting a known template revision, not merely reverting application code. I'm not sure which ownership model fits your organizati

Most Kubernetes advice assumes you have a platform team: specialists who own upgrades, ingress, security policies, and the 2 a.m. pages. The teams I am writing for usually have three to ten engineers, one of whom “knows Kubernetes,” and no dedicated platform team. They depend on a cluster that nobody fully owns. I work in enterprise environments where platform teams are large and everything is process. This article is the opposite exercise: what is the minimum discipline a small team needs to run Kubernetes in production—and what enterprise baggage should it refuse to copy? The question that matters more than any tool Before any checklist: who owns the platform after the migration is finished? Not “who set it up.” Who owns upgrades next year, certificate renewals, the CNI version, and deprecated APIs? If the answer is one person's name, you do not have a platform. You have key-person risk with YAML on top. If the answer is “nobody, really,” Kubernetes is invisible operational debt accumulating interest. The rest of this checklist exists to make that ownership small enough for a small team to carry. For each item, score 0 if it does not exist, 1 if it exists but is informal or untested, and 2 if it is documented and tested. The purpose is not to produce a flattering number. It is to expose the next few conversations the team needs to have. 1. Deployments: Git is the source of truth Treat Git as the source of truth for workloads and cluster configuration, including temporary fixes. Use one reconciliation path—for example, Argo CD or Flux—so production changes are reviewed and reproducible. Keep emergency access, but reconcile every emergency change back into Git. Define and test a rollback path for every service. A Git revert is useful only if your delivery process can deploy it safely. This converts your cluster from a mystery into a diff. Every other practice gets easier once “what is running?” has an answer. 2. The rollout basics that prevent late-night incidents R

If you love the simplicity of Docker Swarm (native Compose files, lightweight single binary) but miss the advanced capabilities of Kubernetes (targeted label placement, SRE-grade observability, built-in DNS service discovery, and zero-trust ingress), meet Gubernator (gbnt) . We are excited to release Gubernator v2.13.0 , introducing three massive feature suites natively integrated into a single binary and a modern Material Design 3 Flutter Web Dashboard: Google SRE Multi-Burn-Rate SLO Engine & Interactive Suite CoreDNS 4-Tab Management Suite & Interactive Dig Playground Caddy Ingress & Zero-Trust Reverse Proxy Suite Fun Fact: The entirety of Gubernator's codebase, multi-node deployment pipelines, and SRE features were designed, built, and pair-programmed using **Google Antigravity (AGY) , Google DeepMind's agentic AI coding assistant! Let's dive into what's new and how you can level up your self-hosted or production container clusters! 1. Google SRE Multi-Burn-Rate SLO Engine & Web Suite Defining Service Level Objectives (SLOs) and tracking Error Budgets is the gold standard of Site Reliability Engineering. Until now, implementing SLOs meant running heavy Kubernetes CRDs (via tools like Sloth or Pyrra) or using costly SaaS platforms. Gubernator v2.13.0 brings Google SRE Workbook (Chapter 5) compliant multi-burn-rate alerting straight to simple docker-compose.yml services: version : " 3.8" services : payment-api : image : hashicorp/http-echo:latest labels : gbnt.slo.enable : " true" gbnt.slo.target : " 99.9" gbnt.slo.window : " 30d" gbnt.slo.template : " caddy-http" gbnt.slo.journey : " Checkout Flow" What makes Gubernator's SLO Suite unique? Google Multi-Burn-Rate Alerting : Automatically generates standard 4-window Prometheus recording and alert rules ( Critical Page 1h/6h & Warning Ticket 3d/14d ). Dynamic "No-Code" Management : Click "+ Configure / Add SLO" in the Web UI or call POST /v1/slo/edit to create, edit, or disable SLOs on the fly without editing Compose

Use a kill switch inside the checker when your uptime probes start amplifying an incident — one feature flag, read on every tick, that can disable the noisy checks and stop the retries at the source. Reach for tuned backoff and jitter instead when the retry storm stays inside a single process and never fans out onto a dependency somebody else is paging for. Both are cheap to build. Only one of them lets you quiet a polling client while its target is already on fire. I run cron and queue infrastructure, so most of my pages arrive as either "the job didn't run" or "the job ran four times." Health checking sits in the same family of problems: a small, frequent, automated request that multiplies badly when something upstream changes shape. What follows is the runbook I settled on after a fleet of pollers turned a non-incident into a real one — the failure mode, where the switch belongs, the implementation, and how to verify the flip before you walk away from the terminal. What actually turns a polling client's uptime checks into a retry storm? Amplification. A single check is one request every 15 or 30 seconds, which nobody notices; a fleet of checks with retries layered on top is a synchronized load generator pointed at whatever you decided was important enough to monitor. The math is unkind. Take 40 instances, a 5s interval, and 3 retries per failed attempt, and a dependency that normally handles a trickle of health traffic suddenly sees a couple thousand requests a minute — all of them arriving at the exact moment it's least able to absorb them. Retries stack on top of the polling interval rather than replacing it, and because every poller sees the same failure at the same time, they all back off together and return together. The Google SRE book calls out this shape under cascading failures, and the load pattern that comes out of it looks nothing like organic traffic: sawtooth spikes, perfectly aligned, growing until something sheds load. The worst one I've dealt wit

TL;DR: Grafana Agent reached End-of-Life on November 1, 2025 and has been replaced by Grafana Alloy. Alloy consolidates Agent's Static mode, Flow mode, and Kubernetes Operator into a single collector built on the OpenTelemetry Collector while maintaining native support for Prometheus and Loki. If you're using Flow mode, migration is relatively straightforward. If you're using Static mode, the migration process will involve reviewing and testing the converted configuration. Before switching over, verify relabeling rules, recheck resource usage, and confirm that Prometheus and Loki are receiving the same data and labels as before. If you're still running Promtail, it's worth migrating both to Alloy at the same time since Promtail is also End-of-Life. If you deployed Grafana Agent a couple of years ago, there's a good chance you haven't thought about it since. It quietly collects metrics, ships logs, and generally stays out of the way. What you may not realize is that Grafana Agent reached End-of-Life on November 1, 2025. That includes Static mode, Flow mode, and the Kubernetes Operator. Grafana Labs has stopped creating bug fixes, security patches, and official support. If you're still running it, your collection layer is probably still performing normally, but is now unsupported. That doesn't necessarily mean it will stop working tomorrow, plenty of unsupported software continues running for years. It does mean you're taking on the risk yourself, especially as the rest of your monitoring stack continues to evolve. This article covers why Grafana Labs replaced Agent with Alloy, what actually changes during the migration, and where people tend to run into problems. Why Grafana Agent was deprecated One of the biggest issues with Grafana Agent is that it was essentially three agents, not one product: Static mode, which used YAML and looked similar to Prometheus. Flow mode, which introduced a component-based configuration using River. The Kubernetes Operator, which manage

Series: Platform engineering with Go | Topics: Go, Kubernetes, Cobra, client-go, metrics-server, Platform Engineering This is part of the Platform Engineering with Go series. This post builds on the Cobra CLI patterns from post 4 and client-go from post 3. Read post 4 first if you haven't yet. kubectl top tells you what's happening. It doesn't tell you how close to the edge you are. In post 3 and post 4 , we built a health reporter and learned how to structure a Go CLI with Cobra. Now we put both together into something with real operational value. kubectl top pods -n production NAME CPU ( cores ) MEMORY ( bytes ) go-api-7d6b9f8c4-xk2pq 240m 490Mi go-api-7d6b9f8c4-mn9rt 180m 210Mi go-api-7d6b9f8c4-p8wvz 200m 198Mi That first pod is using 490Mi of memory. Is that fine or is that a problem? Without knowing the limit, you can't tell. You'd have to run kubectl describe pod go-api-7d6b9f8c4-xk2pq , find the resources section, do the mental arithmetic, and repeat for every pod you care about. ferctl top does all of that in one command: ferctl top -n production NAMESPACE NAME CPU USE CPU REQ CPU LIM CPU% MEM USE MEM REQ MEM LIM MEM% STATUS production go-api-7d6b9f8c4-xk2pq 240m 250m 500m 48% 490Mi 256Mi 512Mi 95% !! CRITICAL production go-api-7d6b9f8c4-mn9rt 180m 250m 500m 36% 210Mi 256Mi 512Mi 41% OK production go-api-7d6b9f8c4-p8wvz 200m 250m 500m 40% 198Mi 256Mi 512Mi 38% OK One pod is at 95% of its memory limit. In production, that's a page waiting to happen. ferctl top catches it before it becomes an incident. What you'll learn How to extend the Cobra CLI structure from post 4 with a real subcommand How to query the metrics-server API using k8s.io/metrics How to correlate live metrics with pod specs to show usage vs limits How to implement configurable near-limit warnings How to format clean aligned output with tabwriter How to verify the tool against your real minikube cluster Prerequisites Posts 1–4 read; client-go patterns from post 3 , Cobra CLI structure from pos

Originally published at wostal.eu . TL;DR : My Hetzner k3s lab quietly became a platform. Dozens of operators with leader-election leases hammered the default datastore — SQLite via kine — until compaction entered a death-spiral: 1.36M rows, a 13.8 GB WAL that wouldn't checkpoint, CPU pinned at 99%, load average 79 on 8 cores. I stopped the bleeding by truncating the WAL, then migrated the control plane to embedded etcd (7.5 GB SQLite → 313 MB etcd, load 79 → 5). This is the full postmortem — and the lessons. This is a war story, not a tutorial. It's about the moment a homelab stops being a homelab and starts behaving like production — without ever announcing it. The cluster in question, homelab , is the Hetzner k3s setup I wrote about previously . It started small. It did not stay small. In this post I'll cover: How an overgrown lab broke the default datastore — the kine/SQLite compaction death-spiral The firefight — measuring instead of guessing, and the fix that actually worked The permanent fix — migrating the control plane to embedded etcd, and the honest caveats The meta-lesson — how to recognize when your lab has become a platform A diagnostic runbook — so next time it's minutes, not hours There's a companion piece to this incident. The CI pipeline that ran this etcd migration was itself freshly — and badly — migrated, and debugging it cost me hours over a single missing newline. I split that into its own post: I Let an AI Re-Platform My CI Pipeline. Here's What Broke. Context: it's "just a homelab" — except it isn't homelab began like any homelab: one k3s node on Hetzner, a few things to play with. The problem is that over months it quietly became a platform . A single master node ( cx43 , 8 vCPU / 16 GB, untainted, and also carrying Longhorn and workloads) now runs: ArgoCD, Kargo, Crossplane/Upbound, CloudNativePG, EMQX, Longhorn, trivy-operator, kubescape, Gatekeeper, Goldilocks/VPA, VictoriaMetrics, Loki, OpenTelemetry, Argo Workflows/Events/Rollouts, kga

Autonomous agents don't just fail loudly—they fail expensively. A single misconfigured retry loop between an agent and an LLM can generate thousands of redundant tool calls and API requests before anyone notices, turning a minor logic bug into a five-figure cloud bill. PolicyAware is built to be the operational safety net that catches this class of failure before it reaches your finance team's dashboard. 1. The Recursive Agent Crisis Every SRE and platform engineer who has run agentic workloads in production has a version of this story. An agent is wired to call an LLM, interpret the response, and take an action—often invoking another tool, which produces output that gets fed straight back into the same LLM. Under normal conditions this loop terminates in a few steps. Under a bad prompt, a malformed tool response, or a subtle logic error, it doesn't. The agent gets stuck reasoning in circles: it calls a tool, receives an ambiguous or malformed result, decides the task is incomplete, and calls the LLM again to "retry." Each retry consumes tokens, each tool call hits a downstream API, and there is no natural circuit breaker unless one has been explicitly engineered. Within minutes, a single stuck session can produce: Thousands of duplicate or contradictory API calls to internal and third-party services. Sustained LLM token consumption that dwarfs normal daily usage. Cascading load on downstream systems that were never designed for machine-speed request volume. By the time monitoring dashboards catch the anomaly—if they catch it at all—the damage is already done: a runaway bill, a rate-limited API partner, or a compromised production database from thousands of unchecked write attempts. Traditional APM tools tell you a service is under load; they don't tell you an autonomous agent is the one generating that load, or why. This is why the recursive agent crisis is fundamentally a governance problem, not just a monitoring problem. Rate limits and cost alerts fire after the

Your monitoring catches problems on everything except itself. Here is how an always-firing Watchdog alert plus an external heartbeat check turns silence into a signal, so you find out when your own alerting dies. TL;DR A monitoring system can't reliably monitor its own failure, so use a dead man's switch. Create an always-firing Prometheus Watchdog alert and route it to an independent external heartbeat service. As long as the monitoring pipeline is working, the Watchdog continuously refreshes the heartbeat. If Prometheus, Alertmanager, or the delivery path fails, the heartbeat stops and the external service alerts you through a separate channel. The key is independence: the system responsible for detecting that your monitoring is down must not depend on the monitoring stack itself. One of the traps of creating alerts on a monitoring stack is the hidden assumption that the mechanism evaluating the alert is running properly and has the ability to evaluate it. Prometheus watches your hosts and Alertmanager delivers the warnings. But what watches Prometheus? If something goes wrong and the monitoring stack fails in the middle of the night, no alerts are going out but there is definitely a problem. That is the failure mode that you should be most concerned about, because it is the one your monitoring cannot report on. The fix is an old idea with a grim name: a dead man's switch. A train's dead man's switch stops the train when the operator stops holding it down. The safe state requires continuous positive action, while the absence of that action is what triggers the response. Applied to monitoring, it means building one alert whose silence is itself the alarm. Step one: an alert that always fires This feels backwards the first time you see it, and it took me a little time to get it right. Basically, you create an alert with a condition that is always true, so it fires constantly, forever, on purpose. In the Prometheus world this is conventionally called Watchdog. - aler