标签：#devops

Most "I built a home SOC" posts show you the pretty dashboard and stop there. This one is about what happened after the dashboard loaded — because that's where the real learning was. 🖥️ The setup Three Mac Minis, one control machine, all Ubuntu. Mac Mini 1 runs the Wazuh manager, indexer, and dashboard — the security monitoring core. Mac Mini 2 handles AWS IAM permission scanning (Cloudsplaining, ScoutSuite). Mac Mini 3 is a documentation hub plus a custom vulnerability-intelligence pipeline pulling from the NVD API and NIST's CPRT compliance framework data. A fourth machine, my daily driver, orchestrates all three over SSH. Today's goal: get every machine reporting into Wazuh, so I'd have real, fleet-wide visibility — not just a single monitored box. 🐛 Bug #1: the silent feed stall After installing the Wazuh agent on Mac Mini 3, vulnerability scanning just... sat there. The log showed Initiating update feed process and then nothing. No error, no timeout, no progress. I ruled things out methodically: DNS resolved fine, curl could reach Wazuh's CTI feed endpoint and complete a full TLS handshake, disk space was nowhere close to full, and there were no stale lock files sitting around. Every individual piece of the pipeline worked in isolation — but the feed download itself never completed. The actual fix: upgrade Wazuh from 4.14.5 to 4.14.6. A newer point release fixed whatever was silently stalling the content-updater module. Once upgraded, the feed downloaded, decompressed, and the scanner came alive within minutes. Lesson: when every individual network/system check passes but a specific module still won't progress, check for a version-specific bug before going deeper down the debugging rabbit hole. 🔒 Bug #2: locking myself out with fail2ban Mid-troubleshooting, my control machine's SSH connection to Mac Mini 3 started getting flatly refused — no prompt, no error, just Connection refused. Ping worked fine. Every other port responded. Just port 22, silently dead. fai

* When a project first starts, CI/CD pipelines are usually simple. * Build the application. Run a few tests. Deploy somewhere. Done. Then six months pass. Another test gets added. Then another deployment step. A security scan. Performance tests. Notifications. More environments. Before long, a pipeline that once took five minutes now takes forty-five. I've seen this happen more than once, and it's rarely because Azure DevOps is the problem. It's usually because nobody ever stops to ask one simple question: Does this step still belong here? Everything Ends Up in the Same Pipeline One of the most common mistakes I see is trying to make a single pipeline do everything. Every Pull Request ends up running: Every unit test Every API test Hundreds of UI tests Security scans Deployment steps Report generation The result? Developers wait longer for feedback, releases become slower, and people eventually start ignoring failed pipelines because they happen too often. Fast Feedback Wins Not every test needs to run on every commit. A better approach is to think about the purpose of each pipeline. For a Pull Request, I want answers quickly. That usually means: Build the application Run unit tests Run a small smoke test suite Stop if something important fails Everything else can happen later. Long-running regression tests, cross-browser testing and other expensive checks are often better suited to scheduled or nightly pipelines. Pipelines Should Evolve A pipeline isn't something you build once and forget about. Every few months it's worth reviewing it. Ask yourself: Which step takes the longest? Which tests fail most often? Are there any tasks nobody remembers adding? Are we getting useful feedback, or just more output? Removing unnecessary work is just as valuable as adding new automation. Final Thoughts Azure DevOps is an incredibly powerful platform, but even the best tools become frustrating if they're overloaded with unnecessary work. The goal isn't to build the biggest pipel

From Zero to a Working EKS Pipeline: Terraform, Ansible, and GitLab CI/CD (and Everything That Broke Along the Way) I recently built an end-to-end deployment pipeline on AWS EKS using Terraform for infrastructure, Ansible for configuration, and GitLab CI/CD to tie it all together. On paper, that sentence sounds clean. In practice, it took several rounds of "why is this failing" before it actually worked. This post is not a "here's how EKS works" tutorial. There are plenty of those. This is the version with the failures left in, the quota limits, the IAM permission walls, the pods that wouldn't schedule, and the resources that refused to die. If you're building something similar, I'm hoping this saves you a few hours of confused Googling. Repo: gitlab.com/nenyeonyema/terraform-eks-ansible-cicd What I Was Building The goal was a full IaC-driven pipeline: Terraform to provision the EKS cluster and supporting AWS infrastructure (VPC, node groups, IAM roles) Ansible to handle configuration tasks on top of the provisioned infrastructure GitLab CI/CD to automate the whole thing — plan, apply, configure, deploy — on every push Simple enough in theory. Four separate blockers said otherwise. Blocker #1: Free Tier ASG Restrictions The first wall I hit was with the Auto Scaling Group for my EKS node group. AWS Free Tier limits how much compute you can provision, and my initial node group sizing quietly ran into those limits — the kind of failure that doesn't always throw an obvious, single-line error. Fix: I resized the node group to stay within Free Tier boundaries and got explicit about instance types and desired/min/max capacity in Terraform, instead of leaving Auto Scaling to make assumptions I couldn't afford. Lesson: If you're building on Free Tier, hardcode your capacity expectations early. Don't let the defaults surprise you later. Blocker #2: EKS Private Endpoint Access By default, EKS clusters can be configured with private-only API server endpoint access. That's grea

GitHub's July 13, 2026 changelog lists REST API management for secret scanning custom patterns. That makes a reviewed configuration-as-code workflow possible. Primary source: GitHub Changelog archive, July 2026 . Follow the July 13 entry to the current REST documentation before implementation. The transport below is an unexecuted design. Endpoint paths, payload fields, permissions, pagination, and plan availability must come from the linked official API reference—not from guessed examples. Define the sync contract A safe synchronizer should: read desired patterns from version control; fetch the remote collection; match each pattern by a stable identity; emit create, update, unchanged, and delete actions; refuse deletion unless explicitly enabled; apply only after the plan is reviewed. patterns.json -> normalize -> diff remote -> plan.json -> approval -> apply Keep API-specific payloads opaque to the diff engine: { "patterns" : [ { "stableKey" : "internal-service-token-v1" , "remoteId" : "SET_AFTER_CREATION" , "payload" : { "REPLACE_WITH_DOCUMENTED_FIELD" : "REPLACE_WITH_REVIEWED_VALUE" } } ], "allowDelete" : false } Placeholders are deliberate. A secret detector's regex fields and matching semantics are security contracts and should never be invented from a blog post. Build a deterministic planner export function plan ( desired , current ) { const remote = new Map ( current . map ( x => [ x . id , x ])); const changes = []; for ( const item of desired . patterns ) { if ( ! item . remoteId ) { changes . push ({ action : " create " , key : item . stableKey }); continue ; } const found = remote . get ( item . remoteId ); if ( ! found ) throw new Error ( `Missing remote pattern ${ item . remoteId } ` ); const same = JSON . stringify ( canonical ( found )) === JSON . stringify ( canonical ( item . payload )); changes . push ({ action : same ? " unchanged " : " update " , key : item . stableKey }); remote . delete ( item . remoteId ); } for ( const orphan of remote . valu

Every "build an AI app" tutorial stops at the demo. Prompt goes in, response comes out, ship it. Nobody covers the part where that demo has real users and you're staring at a Gemini or OpenAI invoice trying to figure out which feature did that. I've spent the last several months building the AI layer for a consumer app that fires vision and language calls on almost every user action. Not a chatbot getting occasional traffic. A product where the model call basically is the product. Here's what I actually had to build, in the order I had to build it. Four problems, not one Cost first, obviously. Tokens are metered, and past a certain volume, calling the model on every request means paying for answers you already gave someone five minutes ago. Latency next. A cache hit lands in milliseconds. A cold model call takes seconds. Users feel that, especially anything camera-driven where they're staring at a loading spinner over their own kitchen counter. Reliability too. Your provider will have an outage or a degraded day at some point. Not if, when. And blast radius. One bug, one bot, one traffic spike, and a $50/day bill becomes a $5,000/day one while everyone's asleep. You don't see any of this in a demo. It shows up with real traffic, and by then it's a lot more expensive to fix than it would've been to build right the first time. Cache on what the query means, not its exact string Key your cache off the literal request text and you've built something close to useless. "What can I make with chicken and rice" and "chicken and rice, what should I cook" mean the same thing and share almost no characters. So embed the query, run a vector similarity search against everything already answered, and if something clears a high threshold, serve that instead of paying for another call. I use 0.95 cosine similarity as the bar. async function checkSemanticCache(embedding: number[], taskType: string, threshold = 0.95) { const { data } = await db.rpc("find_similar_response", { query_emb

Digibee opens with a clear disclaimer: every team there uses Claude Code. This isn't a take from people who skipped the AI tooling revolution. It's an observation from people who shipped with it and ran into the same wall, repeatedly. That wall is enterprise integration. "Enterprise integration isn't a greenfield challenge. It's a completely different category of work, with completely different failure modes that coding agents weren't designed for." What coding agents are actually good at here They're useful for integration work under a narrow set of conditions: well-documented APIs, one-time tasks, low stakes, nothing in production at risk. A quick script to pull from a public endpoint? Great. A throwaway ETL job? Perfect. The problems start the moment an integration needs to be recurring, reliable, auditable, and maintained by someone other than the person who prompted it. The three structural gaps 1. They start from scratch every time. Pre-built connectors for enterprise systems like SAP, Salesforce, or NetSuite encode years of accumulated knowledge — how sequencing works, how idempotency is handled, where the quirks are. A coding agent reasons through all of that fresh on every run. It also suffers from the "lost in the middle" effect: when documentation gets long, LLMs drop content from the middle of their context window and fall back on training data. The more obscure the API, the more likely the generated code quietly fails under real load — not on deployment, but six months later when the CIO notices corrupted records. 2. They produce code, not infrastructure. Integrations need retry logic, failure recovery, credential management, audit trails, monitoring, and alerting. Coding agents produce none of that. You can prompt your way around it piecemeal — but now you're maintaining the integration and five hand-rolled infrastructure components. An agent optimised to iterate fast isn't optimised to fail safely. In production, a bad write means unprocessed payments

Database branching is one of the best ideas serverless Postgres brought to the mainstream. Fork the database at a point in time, get an isolated copy with all the data, run something risky against it, throw it away. It made preview databases and safe migrations feel routine. But a real application is not just a database. It is a database, plus the files it stores in object storage, plus the backend code that serves it, plus, increasingly, the model and gateway config it calls for AI. When you branch only the database, those other three stay shared. Your "branch" points at the same S3 bucket, the same deployed backend, and the same AI configuration as everything else. So it is half a copy, and the half it leaves out is where a lot of the interesting bugs and the scary migrations live. Neon's platform preview changes what a branch contains. A branch now forks the database and its data, the object storage and its files, the functions that run your backend, and the AI gateway config, all at the same point in time, all isolated. A branch stops being a database copy and becomes a whole environment. To make sure that is a real claim and not a diagram, I took a full-stack project, branched it, and checked every layer. Here is what happened. TL;DR Elsewhere, "branch" means the database only. Object storage, backend deploys, and AI config stay shared, so you bolt on scripts to fake per-branch versions of them. A Neon branch forks all four together: Postgres + data, object storage + files, functions (each branch gets its own URL), and the AI gateway. I proved it: branched a project with a DB, a bucket of files, a function, and the gateway. The branch came up with a copy of the rows, a copy of the files on its own storage endpoint, its own function URL, and the gateway. A write to the branch left main untouched, and deleting the branch removed all of it. That makes a branch a real environment: true preview stacks, whole-state bug reproduction, and disposable sandboxes for agent

Most test automation tools look good during a demo. You record a login flow, add an assertion, run it in Chrome, and get a green result. Everyone is impressed. Then the real application gets involved. There are dynamic elements, delayed API responses, test accounts, verification emails, downloaded files, several deployment environments, and a checkout flow that behaves differently on Safari. A few months later, the original test suite has grown from 10 tests to 300. Some failures are product bugs. Others are test problems. A few only happen in CI. Nobody is completely sure which is which. That is when you discover whether you selected a test automation tool or merely a good demo. Creating tests is rarely the main problem When teams compare automation tools, they often begin with questions such as: How quickly can we record a test? Can AI generate the steps? Does it support plain-English instructions? Can a manual tester use it? Does it integrate with our CI pipeline? These are reasonable questions, but they mostly describe the beginning of an automation project. The harder questions appear later: Who updates the tests after a redesign? How do we investigate failures? Can another person understand a test created six months ago? What happens when the original automation engineer leaves? Can we test workflows that involve email, APIs, files, or mobile devices? How much infrastructure do we have to manage? Does the cost increase every time we run the regression suite? The first test tells you whether the tool works. The hundredth test tells you whether the approach works. Maintenance should be part of the evaluation A stable automated test is not a test that never changes. Applications are supposed to change. Buttons move. Components are replaced. Authentication flows evolve. APIs return different data. Product teams redesign entire sections of the interface. The objective is not to prevent tests from changing. It is to make those changes inexpensive and understandable.

Twelve pricing tabs open. Neon, Hetzner, Supabase, Prisma, Scaleway, OVH. My database is half a gigabyte. I was comparing ten-terabyte price curves. At some point this week I typed the words "I am super lost here" about my own infrastructure. I advise companies on this exact class of decision. That sentence still came out of my hands. If you have ever spent an evening deep in provider pricing pages for a workload that fits on a USB stick from 2009, this one is for you. All numbers below come from the live pricing pages as of July 2026. Rates move, so verify before you commit. Three fears, all pointed at the wrong layers I went in worried about getting attacked, running out of space, and being locked in. All three dissolved under ten minutes of honest reading. DDoS lands on the website edge, not the database. My site already sits behind Cloudflare and Vercel, and a database is never publicly exposed. Only the app talks to it. Whichever provider I picked, that attack surface stayed identical. Here is the shape of the stack, and where each fear lives. MANAGED (what I run today) visitors ──> Cloudflare edge ──> Vercel app ──> managed Postgres [DDoS absorbed] [stateless] [never public, app-only access, provider patches, provider backups, provider on-call] SELF-HOSTED (the alternative I priced) visitors ──> Cloudflare edge ──> Vercel app ──> Hetzner CAX11 [DDoS absorbed] [stateless] [Postgres :5432 firewalled to app, SSH hardened, fail2ban + auto- patching = MINE] │ pg_dump every 6h ▼ encrypted ────> Cloudflare R2 [off-site copies] Same edge, same app, same attack surface. Everything in the right-hand box is what changes owners. Storage was a rounding error. My data is 0.5 GB. Even the cheapest self-hosted box includes 40 GB, eighty times headroom before the first extra cent. Lock-in was a phantom too. Managed Postgres is still stock Postgres. Exiting means a dump, a restore, and one connection string change in the deployment environment. Minutes of cutover, no rewrite an

Why Pipelock Is an Egress Agent Firewall, Not an Inbound WAF The question behind the word firewall Security teams hear "firewall" and picture something inbound. A firewall, WAF, or IPS sits in front of a service. Traffic comes from the outside world toward the protected app. The control inspects requests before they reach the app and blocks malicious payloads at the door. That is outside-in protection. It fits web applications, where many attacks have recognizable request shapes: SQL injection, cross-site scripting, known exploit signatures, or malformed protocol behavior. The web server is the thing being attacked, and the attacker sends requests into it. AI agents invert that model. The agent is not only a server receiving input. It reads external content, calls tools, sends HTTP requests, invokes MCP servers, and runs with credentials. The dangerous event is rarely that a hostile packet reached the agent. The dangerous event is that the agent got talked into doing something with outbound effects. That is why Pipelock is built as an egress agent firewall, not a WAF-style inbound firewall. Why inbound filtering is the wrong primary model Prompt injection does not behave like a structured malware packet. It is natural-language instruction sitting in places the agent is supposed to read: a web page, a ticket, a search result, a tool response, an MCP server reply, or a user message. The channel is legitimate. The syntax is often normal. The attack is semantic and context-dependent. Solving that by filtering every input before it reaches the agent turns into an enumeration problem. You write patterns for "ignore previous instructions," then the attacker rephrases. You block one formatting trick, then the instruction is split across paragraphs, hidden in quoted text, encoded, or dressed up as policy text. Known phrases are worth catching, and Pipelock catches known injection markers in content it mediates, but input filtering cannot be the center of the security model.

The Hidden Cost of Manual IAM Review Most teams don't track how long they spend reviewing IAM policies. When I started measuring it on my own team, the numbers were worse than I expected. A thorough manual review of one IAM policy takes 10 to 15 minutes. Not a quick scan. A real review: read every statement, trace every cross-account trust, verify every condition key, check for privilege escalation paths, confirm the resource ARNs match what you think they should. At 4 engineers touching IAM once a week, that's 4 hours a month. 48 hours a year of senior engineers reading JSON documents. And that's the optimistic case. Add a security incident. Add an audit. Add the emergency Friday-afternoon policy change that needs review before deploy. The real number is higher. What manual review misses The problem isn't just the time. It's that humans are bad at repetitive structured-data review, especially under time pressure. Here are the things I've seen slip through manual IAM reviews on production systems: iam:PassRole with no condition. This is the big one. PassRole lets a principal pass a role to a service — and if there's no iam:PassedToService condition, that role can be passed to any service that accepts roles. Including services the attacker controls. The reviewer saw the action, mentally categorized it as "role stuff," and moved on. It was statement 47 of 52 — the reviewer had already been reading policies for 40 minutes. Wildcard resource with sensitive actions. s3:* on Resource: "*" is obvious. s3:GetObject on "arn:aws:s3:::*-backup/*" with a wildcard in the bucket name — that's subtle. The reviewer reads it as "restricted to backup buckets" and moves on. But the wildcard means any bucket ending in -backup , including ones in other accounts if cross-account access is configured. Missing aws:SourceArn on Lambda invocation permissions. When you grant another service permission to invoke your Lambda function, you need aws:SourceArn to prevent the confused deputy

For a long time, teams treated browser test reliability as a framework problem. When tests failed, the usual response was to change selectors, add waits, increase retries, or replace one automation library with another. That approach made sense when the main challenge was simply controlling a browser. Modern applications are different. A single user journey may now include an identity provider, multi-factor authentication, a streaming AI response, a background API request, a feature flag, a canary deployment, and a frontend rendered differently across several operating systems. The test framework is still important, but it is only one part of the reliability problem. The bigger question is whether the entire testing system gives the team enough evidence to make a release decision. Headless failures are usually a symptom, not the real problem A common example is a test that passes locally but fails only in headless Chrome. It is tempting to assume that headless mode is simply unreliable. In practice, the difference is often caused by viewport size, rendering behavior, animation timing, fonts, resource loading, or elements being positioned differently when no visible browser window exists. This breakdown of why browser tests fail only in Chrome headless is useful because it separates several failure categories that are often grouped together as “timing issues.” That distinction matters. A test that fails because an element is outside the viewport needs a different fix from a test that fails because a network request completes later in CI. Adding a longer timeout may hide both problems temporarily, but it does not make the test more trustworthy. Retries can make a weak test suite look healthy Retries are one of the easiest ways to reduce visible failures in CI. They are also one of the easiest ways to hide instability. A flaky test that passes on its third attempt still consumed runner time, delayed feedback, created extra logs, and made it harder to determine whether

Every DevOps engineer has done this dance: you've got a chunk of YAML or a Terraform file that looks right, something's rejecting it, and you want a fast sanity check. So you paste it into some random online validator — and a small voice asks, wait, where did that config just go? That config often has structure, comments, sometimes internal hostnames or resource names in it. Pasting infrastructure definitions into an unknown server is a habit worth breaking. So I built a set of validators that never send your config anywhere — they run entirely in your browser. What they are Free, browser-based validators for the formats DevOps folks paste-and-pray most: YAML — catches the indentation and structure errors that make Kubernetes and CI configs fail with cryptic messages Kubernetes manifests — schema-aware checks beyond "is it valid YAML," so you catch the wrong apiVersion or a misplaced field before kubectl apply does Terraform / HCL — structural validation for the syntax slips that terraform validate flags only after you've context-switched away The one design decision that matters 100% client-side. No upload, no signup, no server round-trip. Your config is parsed by JavaScript running in your own tab — it never leaves your machine. You can literally open dev-tools, watch the network panel, and see nothing go out. Turn off your wifi and they still work. This isn't a privacy gimmick — it's the correct architecture for a tool that handles infrastructure definitions. A validator has no business seeing your config on a server it doesn't need to. Why I bother Two reasons, honestly. One: I kept wanting this exact thing and kept not trusting the options. The nth time I hesitated before pasting a manifest into a stranger's website, I decided to just build the version I'd trust. Two: fast feedback loops are the whole game in this job. The gap between "save the file" and "find out it's malformed" is pure friction — and the tighter that loop, the less of your working memory it b

The Linkerd community has announced the release of Linkerd 2.20, introducing a series of performance, observability, and traffic management enhancements that further strengthen the CNCF-graduated service mesh's position as a lightweight alternative for Kubernetes networking. By Craig Risi

Downloading an installer and immediately executing it as root collapses three operational decisions into one command: Which artifact? -> Did these bytes arrive intact? -> Should this host execute them? Separate those decisions and the install becomes reviewable, reproducible, and recoverable. A concrete source-review boundary At commit c58bcd4 , the MonkeyCode runner installation template selects x86_64 or aarch64 , checks AVX on x86, requires root, and downloads an architecture-specific installer before executing it. The reviewed template uses curl -4sSLk , so certificate verification is disabled by -k . It also downloads an unversioned path. I could not find a pinned version, digest, or signature check in that template. That is a statement about controls visible in one pinned file—not a claim that the release service is compromised or that no external release control exists. Put a manifest before execution For each release artifact, publish immutable metadata through a separately protected release process: { "version" : "1.2.3" , "architecture" : "x86_64" , "file" : "runner-installer-1.2.3-x86_64" , "sha256" : "<64 lowercase hex characters>" , "size" : 18439210 , "rollback" : { "previous_version" : "1.2.2" , "artifact" : "runner-installer-1.2.2-x86_64" } } SHA-256 detects bytes that differ from the manifest. It does not prove who authored the manifest. Serve the manifest over validated TLS, pin it through deployment configuration, or sign it and verify the signature with a trusted offline public key. Verify as an unprivileged staging step The companion verify-installer.mjs checks filename, exact size, digest, version, architecture, and rollback metadata: node verify-installer.mjs release-manifest.json fixture-installer.sh node test-verifier.mjs Expected output uses the fixture's actual digest: PASS 1.2.3-fixture sha256=<digest> PASS verified fixture; rejected tampered artifact before execution The negative test appends a line to the artifact and requires both size

The failures that cost me the most in three years of self-hosting were never the ones that threw an error. An error is a gift: it tells you where to look. The expensive ones are the failures that report success while being broken . A page that returns 200 OK . A healthcheck that says the container is fine. A backup that exits cleanly. A command that prints nothing wrong. Everything green, everything lying. Here are four of them, all from the same box (a 2016 desktop, i7-6700 / 32 GB, Docker behind Caddy, reachable only over Tailscale). Each fails by handing you a success signal. Each cost me an evening the first time. The fixes are boring once you know them, the point is knowing the failure exists. Sanitized skeleton with all the config at the end. 1. A loading page that returns 200 This one I could find nothing written about, so it cost me the most. To keep the box quiet, I run the heavy services on-demand: Sablier stops idle containers and starts them on the first request. Caddy (with the Sablier plugin) gates a virtual host behind a container group, serves a "please wait, starting up" page while the group boots, then proxies through: myhost . my - tailnet . ts . net : 8081 { route { sablier http :// sablier : 10000 { group office session_duration 30 m } reverse_proxy nextcloud : 80 } } I gate my whole Nextcloud vhost, WebDAV included, this way. And here is the silent failure: if the gated group is not healthy, Sablier serves that HTML loading page for every request, and it serves it with 200 OK . A browser shows a spinner, fine. But my Obsidian vault syncs over WebDAV, and a WebDAV client asking for a directory listing got a 200 with a chunk of HTML instead of the XML it expected. Sync died with a cryptic no root multistatus found . Nextcloud itself was up and perfectly healthy the whole time. Every uptime check I had was green, because the gate in front kept answering 200 . The structural lesson: the moment you put a service on-demand behind a reverse proxy, tha

If you ask me what part of the system I protect the most, the answer is the database. I've been writing software alone for twenty-four years, and across every platform I've built, the rule has stayed the same: the web servers can take whatever you throw at them, the batches can be rebuilt, but the database has to stay idle on purpose. Not because I love idle databases, but because the day a database actually starts to struggle is a day with very few good options. This article is about what "keep the database idle on purpose" actually means in practice, and about one particular kind of graph that, in my experience, almost nobody is watching. The three layers and what each of them gets I think of a production system as having three tiers, and each tier gets a different rule. The web server tier can be horizontally scaled. If load grows, you add machines. If something is wrong, you take a machine out of the pool, and the others handle it. Failures here are visible immediately, and they're cheap to recover from. The batch server tier can be scaled up or out depending on the work. A batch that's too slow can be split. A batch that crashes can be retried. End users don't see batch servers, so a stuck batch is a problem for me and not for them. Some headroom up here is fine. The database tier is the one I treat completely differently. The database is not where you absorb load. The database is what you protect from load. The reason is simple: the other tiers can be rebuilt or re-scaled. The database is the irreplaceable record. If it slows down, everything slows down. If it falls over, you don't have many minutes before the rest of the stack notices. So my rule for the database is: keep it idle. Not idle in the sense of "doing nothing." Idle in the sense of "running well below its capacity, at all times, so that any extra load it picks up has somewhere to go." For more than a decade I ran a large appliance-grade database where I kept the load average below 1 at all times. N

Python is an excellent language for automating cloud infrastructure, but the official Kubernetes Python client leaves developers with an important architectural decision: Where should Kubernetes manifests live? Should they be constructed directly with Python objects? Embedded as multiline strings? Or stored as external files and rendered at runtime? Each approach works, but they have very different implications for readability, maintainability, and long-term operational cost. The key is recognizing that deployment logic and platform configuration evolve on different lifecycles. Your deployment code, the part that authenticates to Kubernetes, renders templates, and applies resources, may remain unchanged for months. Your manifests, however, often change weekly as applications evolve, resource limits are tuned, cloud-provider annotations are added, or networking requirements change. When those two concerns are tightly coupled, even a configuration, only change forces you to modify, test, and redeploy the delivery or application code itself. Over time, this increases maintenance costs, slows platform changes, and makes configuration drift and production mistakes more likely. This is a familiar software engineering principle: separate concerns that evolve independently. The same thinking that keeps application configuration separate from executable code also applies to Kubernetes manifests. Treating manifests as first-class configuration artifacts allows them to evolve independently from the Python code that delivers them. In this article we'll compare three ways of deploying Kubernetes resources with the official kubernetes-python-client , ranging from tightly coupled implementations to a design that cleanly separates deployment logic from platform configuration. The Landscape at a Glance The comparison below assumes a common application deployment scenario, where the desired state is largely known ahead of time. Controllers and Operators have fundamentally different r

Here is a design decision we made early, wrote into the architecture as an invariant, and have refused to revisit since: our agent accepts no commands. Not "we don't currently use that feature" — the hub has no way to tell an installed agent to do anything at all. No remote execution, no self-update, no "collect this for us right now". It sends data outward, and that is the entire surface. This is not a limitation we are working around. It is the product. And it costs us features that customers ask for, which is exactly why it is worth explaining. The uncomfortable arithmetic of remote control Any tool that can update a plugin across fifty client sites is, by construction, a tool that can execute code on fifty client sites. Any dashboard that can restart a service on your server holds, somewhere, a credential that lets it in. This is not a flaw in those products — it is what they are for. You cannot automate a repair without the power to perform it. But that power has an owner, and the owner has a login, and the login has a support team, and somewhere in that chain there is a version of the software with a bug in it. When the tool is compromised, the blast radius is not the tool. It is every machine the tool could reach. The industry has already run this experiment at scale. In July 2021, attackers exploited a vulnerability in a widely used remote monitoring and management platform. They did not break into a single company — they broke into the thing that had access to the companies. Roughly sixty managed service providers were hit, and through them, an estimated 800 to 1,500 downstream businesses were encrypted in a single weekend, with a $70 million ransom demand attached. Read that shape again, because it is the whole argument: the victims did nothing wrong. They had bought a well-known product from a serious vendor and installed it exactly as instructed. Their compromise arrived through the door they had deliberately, sensibly, contractually left open — the one

Enhancing CI/CD and E2E Testing with Sentry Integration in tvview TL;DR: I integrated Sentry for error tracking and improved End-to-End (E2E) testing in the tvview project, enhancing the CI/CD pipeline. This resulted in a score increase from 85 to 95+. The Problem The tvview project lacked comprehensive error tracking and E2E testing, making it difficult to identify and resolve issues in production. The existing CI/CD pipeline needed improvement to ensure smoother deployments and better code quality. What I Tried First Initially, I focused on setting up E2E tests using Vitest, but encountered issues with the test configuration. I also attempted to integrate Sentry, but faced challenges with the DSN (Data Source Name) configuration. The Implementation Step 1: Configuring Sentry To integrate Sentry, I created separate configuration files for the client, edge runtime, and server: // sentry.client.config.ts import * as Sentry from " @sentry/nextjs " ; Sentry . init ({ dsn : " https://385038c88b6eb6ddac52d05a144ab8c1@o4511628189630464.ingest.us.sent " , // Additional configuration options }); // sentry.edge.config.ts import * as Sentry from " @sentry/nextjs " ; Sentry . init ({ dsn : " https://385038c88b6eb6ddac52d05a144ab8c1@o4511628189630464.ingest.us.sent " , // Additional configuration options }); // sentry.server.config.ts import * as Sentry from " @sentry/nextjs " ; Sentry . init ({ dsn : " https://385038c88b6eb6ddac52d05a144ab8c1@o4511628189630464.ingest.us.sentry.io " , // Additional configuration options }); Step 2: Enhancing CI/CD Pipeline I updated the .github/workflows/ci-e2e.yml file to include Sentry configuration and E2E testing: name : 📺 CI + E2E — TVView on : push : branches : [ main ] workflow_dispatch : {} schedule : - cron : " 35 6 * * *" jobs : build-and-test : runs-on : ubuntu-latest steps : - name : Checkout code uses : actions/checkout@v2 - name : Install dependencies run : npm install - name : Generate Prisma client env : DATABASE_URL : " postgre