标签：#DevOps

Traditional test automation frameworks often carry heavy maintenance costs, slow release cycles, and high knowledge dependency. By transitioning from standard script creation to a governed AI Test Automation Factory , engineering teams can shift their focus from writing boilerplate code to high-value validation and architectural optimization. Here is an architectural breakdown of how multi-agent AI systems, governed telemetry, and Model Context Protocol (MCP) transform enterprise quality engineering. The Problem: The 45-Hour Manual Bottleneck Building a end-to-end BDD automation suite manually requires significant time per user story—often taking up to 45 hours across five distinct steps: Context Generation & Requirements Review (~8 hrs) Manual Test Case Design (~9 hrs) Cucumber Feature File Creation (~8 hrs) Page Object Model Generation (~8 hrs) Step Definition Implementation (~10 hrs) This traditional workflow creates coverage gaps, inconsistent code quality, and defect leakage. The Solution: Multi-Agent AI Automation Pipeline Instead of relying on single prompts, an AI Test Automation Factory routes requirement artifacts (BRDs / User Stories) through specialized agents: [BRD / User Story] │ ▼ [Context Agent] ──► [Test Case Agent] ──► [Feature File Agent] │ [Automation Suite] ◄── [Step Definition Agent] ◄── [Page Object Agent] Context Agent: Parses acceptance criteria and enterprise domain knowledge. Test Case Agent: Auto-generates exhaustive test scenario matrices. Feature File Agent: Drafts standardized BDD Cucumber feature files. Page Object & Step Def Agents: Constructs clean design patterns (POM) and matching step implementations. Measurable ROI: Before vs. After AI By replacing manual generation with agentic workflows, the effort to automate a scenario drops from 45 hours to 9.5 hours: Phase Manual Effort AI-Driven Effort Time Saved Context Generation 8 hrs 2 hrs 75% Test Design 9 hrs 2 hrs 78% Feature File Creation 8 hrs 0.5 hrs 94% Page Object Creation 8 h

The cloud ecosystem in 2026 isn't just about moving VMs to the public cloud—it's heavily driven by hybrid operations, unified security telemetry, AI integration, and complex governance across multi-region architectures. As enterprise tech stacks evolve, Microsoft Azure certifications remain a primary yardstick for technical competence, but knowing which track to target is where most engineers get stuck. As someone who works closely with cloud certification blueprints and enterprise deployments, I wanted to map out where Microsoft credentials stand today, what the market actually demands, and how specific exams fit real-world scenarios. Market Trends: Why Azure Credentials Still Drive Real ROI in 2026 The value of certification has shifted from basic feature recognition to proving operational problem-solving under real constraints. Hands-on Scenario Focus: Exams increasingly test scenario-based trade-offs—balancing performance, cost, and strict security requirements rather than simple definition checks. Role-Based Specialization: Instead of broad, generic tracks, Microsoft continues to refine specialized pathways for developers, security analysts, and hybrid infrastructure specialists. Continuous Free Renewal: Earning the badge is step one, but maintaining active status requires passing annual, open-book renewal assessments directly through Microsoft Learn, ensuring skills don't stall out. Mapping Azure Exams to Real-World Enterprise Scenarios Depending on your daily engineering focus or career targets, here is how the core role-based tracks align with active projects: App Modernization & Cloud-Native Dev: AZ-204 (Azure Developer Associate) The Scenario: Refactoring monolithic legacy apps into containerized microservices using Azure App Service, Azure Functions, and Cosmos DB while setting up secure authentication via Microsoft Entra ID. Hybrid Infrastructure & Server Ops: AZ-800 (Administering Windows Server Hybrid Core Infrastructure) The Scenario: Managing mixed e

Answers from the incident where every dashboard looked politely wrong. The release-api deployment had already been marked complete when the invoice page began returning 503s. The new container was serving traffic, the PostgreSQL migration had committed, and the feature flag was on. A NetworkPolicy added in another repository prevented the new pod from reaching tax-rate-cache . The application team saw errors, the database team saw a clean migration, and Platform saw green nodes. By the time we put all three facts in one incident channel, 63 deployment messages had buried the one that mattered. “Is This Actually A DevOps Failure Or Just One Bad Deploy?” It was a DevOps failure because four teams completed valid local work and nobody owned the release handoff between them. Calling it “just a bad deploy” would have been convenient. We could have fixed the policy, replayed the release, written a short incident note, and carried on pretending that a green Argo CD application means a service is ready for users. The pod was healthy according to Kubernetes. It was also unable to call a dependency required to render an invoice. Both things can be true, which is why a deployment status alone is a fairly poor witness. Our old release process had hidden contracts in too many places: The service repository declared its image and Helm values. The infrastructure repository held network rules. Database migrations ran from a separate GitHub Actions workflow. Feature flags lived in LaunchDarkly, owned by whoever had last touched the feature. The runbook lived in Confluence, where it had last been edited in February. We’ve started putting the release dependencies in the service repository, close to the code that needs them. It is not a clever system. It is a file that a human can read during an incident and a pipeline can check before promotion. release : service : release-api requires : - dependency : tax-rate-cache namespace : finance port : 8080 network_policy : allow-release-api-t

Introduction: Kubernetes can feel overwhelming when you first hear terms like Pods, Services, and Deployments thrown around. In this first post of my Kubernetes series, I’ll break down the fundamentals — what Kubernetes actually solves, and the core building blocks you need to understand before going further. What is Kubernetes? Kubernetes is an open-source container orchestration tool , originally developed by Google. It helps manage containerized applications across different environments — physical machines, virtual machines, and cloud environments — which makes it a great fit for hybrid deployment setups. Why Kubernetes? The Problem It Solves To understand why Kubernetes exists, look at the trend that led to it: Applications moved from monolith to microservices. That shift drastically increased the number of containers teams had to manage. Managing hundreds of containers by hand became unsustainable — teams needed a proper way to orchestrate them. Key Features High Availability — no downtime Scalability — scale up or down based on load and performance needs Disaster Recovery — backup and restore built into the ecosystem Main Kubernetes Components Pods: Abstraction over containers Services: Stable networking & communication Ingress: Routes external traffic into the cluster ConfigMaps & Secrets: External configuration Volumes : Data persistence Deployments & StatefulSets: Replication (stateless vs. stateful) DaemonSets: One Pod per node, auto-scaled with the cluster

"Should we add AI to our product?" isn't really the right question anymore. Most SaaS founders and product teams have moved past whether to add AI and are stuck on how — how to do it without a six-month rebuild, a runaway API bill, or a feature that looks impressive in a demo but nobody actually uses. At Softication Technology Pvt. Ltd., we've worked with SaaS teams integrating AI into products ranging from CRMs to internal tooling to customer support platforms. This guide lays out the practical, engineering-first approach we use — the decisions that actually matter, and the ones that are just noise. Table of Contents Why "Adding AI" Isn't One Thing Step 1: Find the Right Entry Point Step 2: Choose Your Integration Pattern Step 3: Design the Architecture Step 4: Handle Cost, Latency, and Reliability Step 5: Ship Small, Measure, Expand Common Mistakes We See Final Thoughts Why "Adding AI" Isn't One Thing "AI integration" gets used as a catch-all term, but it covers very different engineering problems: Generating or rewriting content Answering questions using your product's own data Classifying, tagging, or routing records automatically Predicting outcomes from historical data Automating multi-step workflows end to end Each of these needs a different technical approach. The biggest mistake teams make is picking a technology (usually "let's use an LLM for everything") before defining which of these problems they're actually solving. Step 1: Find the Right Entry Point Before writing any code, look at your product usage data and support tickets for patterns like: Repetitive manual work — users doing the same categorization, summarization, or data entry over and over Search or discovery friction — users struggling to find information that exists in your product Decision bottlenecks — users waiting on judgment calls that follow a somewhat predictable pattern A good first AI feature is narrow, has a clear success metric, and solves a problem your users already complain abou

I recently worked through a hands-on Amazon S3 practical series covering the features I would actually expect to use while working with AWS storage. Instead of only documenting definitions, this post focuses on what I configured, the commands I used, how I verified the behavior, and what I observed when something went wrong . For the concepts behind these practicals, I have already covered S3 in two detailed sessions: Session 1: AWS S3 Deep Dive — Objects, Encryption, Bucket Policies & Everything In Between Session 2: AWS S3 — Versioning, Static Hosting, CORS, Object Lock & More This post is the practical companion to those two sessions. The concepts are covered there; here I focus on actually building, testing, verifying, and troubleshooting the S3 features. The labs covered: S3 bucket configuration and lifecycle management Bucket policies with IAM, EC2 and HTTPS-only access SSE-KMS encryption with CloudTrail verification Pre-signed URLs AWS CLI s3 sync S3 Versioning and version recovery Static website hosting S3 CORS S3 Object Lock Note: This is a practical write-up, so I have intentionally kept the focus on implementation and verification rather than turning it into a generic S3 theory article. 1. S3 Bucket Configuration and Lifecycle Management Objective Create an S3 bucket with a secure baseline and configure a lifecycle rule that automatically transitions objects to lower-cost storage classes over time. Configuration For the lab: Block Public Access remained enabled. Bucket Versioning was enabled. Lifecycle rule: s3-lab-lifecycle The rule applied to all objects. Current objects transition to: Standard-IA after 30 days Glacier Flexible Retrieval after 90 days The lifecycle flow was: Day 0 ↓ Object uploaded ↓ Day 30 → Standard-IA ↓ Day 90 → Glacier Flexible Retrieval Result The lifecycle rule was successfully created and enabled, and the S3 console confirmed the configured transition periods. What this demonstrates Instead of manually moving old objects, S3 Life

An MCP server can ship a change that breaks every agent calling it, and nothing in your monitoring will notice. The endpoint still answers 200. The initialize handshake still completes. tools/list still returns a result. Every signal a conventional uptime check knows how to read stays green — and an agent that memorized last week's tool contract starts failing anyway, because the contract underneath it moved. This is drift: a server's tool inventory or a tool's input schema changing between two points in time, with no transport-level symptom at all. Drift is not a hypothetical. MCP servers are young, most are maintained by small teams, and a tool's inputSchema is just a JSON object in a deploy — there is no compiler stopping someone from renaming a required field, tightening an enum, or dropping a tool nobody remembered an agent still called. The only way to catch it is to have looked at the server before and remember what you saw. What Actually Counts as Drift Drift is anything about a server's advertised capability contract that differs from the last time you checked. Concretely: A tool disappears. It was in yesterday's tools/list , it is not in today's. Any agent that calls it now gets a JSON-RPC error mid-flight, not at startup — the failure shows up wherever the agent happens to reach for that tool. A tool appears. Informational on its own, but worth recording — it is also how you notice a server quietly forking its capability set per client or per deploy. A tool's contract changes shape. Same name, different inputSchema — a field renamed, a type narrowed, a new required parameter, a changed description that alters how an LLM decides to call it. The tool is still callable, which is what makes this the dangerous case: nothing errors immediately, calls just start failing validation or getting silently misinterpreted. The capability set changes. The server stops advertising resources or prompts , or starts. Anything built against the old capability list breaks the

VRP Is Ready for External Validation — Who Will Be the First to Pilot It? My name is Vitalijus Riabovas. I am the independent architect and creator of VRP — Veil Routing Protocol . VRP is a continuity-first networking architecture built around a simple principle: A logical session should not have to die simply because the network underneath it changed. Wi-Fi → LTE/5G. IP mutation. NAT / CGNAT churn. Temporary blackout. Path failure. Recovery. Replay attempts. Stale authority. Duplicate execution. For a long time, VRP was primarily architecture, runtime engineering and internal validation. That stage has changed. The public validation boundary exists now. And I am inviting serious engineers and organisations to test it. DON'T TRUST MY CLAIMS. TEST THEM. I am not asking the networking industry to believe a presentation. I built the measurement boundary. The public VRP Validation Kit provides engineers with an environment for evaluating observable behaviour independently. You can: clone the repository; run the Docker scenarios; inspect generated evidence; verify manifests and hashes; attack the evidence; delete events; duplicate events; reorder events; attempt replay; introduce stale conditions; corrupt artifacts; run the verifier; reproduce PASS / REJECT / INCOMPLETE outcomes. If you believe something is wrong, try to produce a reproducible contradiction. Give me: environment → scenario → commands → evidence → result That is useful engineering. WHAT HAS BEEN BUILT? VRP has moved far beyond an architectural diagram. The project now includes multiple engineering layers. Continuity architecture Logical session identity is designed to survive changes in the underlying network path. The architecture is being developed around continuity rather than assuming that transport identity and logical session identity must always be the same thing. Runtime The protected runtime implements the private VRP mechanisms. That implementation is not public . State and transition handling T

Um 02:47 Uhr stoppte mein System ein Deployment. Kein Mensch war wach. Es war ein Dienstagmorgen, als mein Guard-System anschlug. Nicht wegen eines fehlgeschlagenen Tests. Nicht wegen eines Syntaxfehlers. Ein Agent hatte versucht, einen Commit zu pushen, der einen AWS-API-Schlüssel enthielt. Der Schlüssel steckte in einer Konfigurationsdatei, die eigentlich nie ins Repository sollte. Der Deployment-Prozess wurde blockiert. Um 02:47 Uhr. Kein Mensch hätte das um diese Zeit gesehen. Der Schlüssel wäre live gegangen. Das war kein Einzelfall. Es war der 47. Vorfall in 14 Monaten, den mein System automatisch abgefangen hatte, bevor er Schaden anrichten konnte. Und er hat mir klarer als je zuvor gezeigt, warum das Regelwerk wichtiger ist als das Modell selbst. Was ein Guard-System wirklich ist Die meisten, die über KI-Sicherheit sprechen, meinen Alignment, Halluzinationen oder Trainingsdaten. Das sind echte Probleme, aber sie liegen auf einer anderen Ebene. Ich rede von etwas Handwerklichem: einem System, das verhindert, dass ein KI-Agent im laufenden Betrieb Fehler macht, die Menschen Geld oder Daten kosten. Mein System läuft auf einem Prinzip, das ich GRIP nenne: Guards, Rules, Isolation, Protocol. Jeder Agent, der in meinem Stack läuft, durchläuft vor jeder kritischen Aktion eine Prüfkette. Nicht als Empfehlung. Als harter Block. Das bedeutet konkret: Der Agent darf nicht weiter, bis das Problem behoben ist. Kein Fallback, kein "try anyway", kein Override ohne explizite Freigabe. # Beispiel: Pre-Commit Guard gegen Secrets #!/bin/bash STAGED_FILES = $( git diff --cached --name-only ) for FILE in $STAGED_FILES ; do if grep -rE "(AKIA[0-9A-Z]{16}|sk-[a-zA-Z0-9]{32,}|ghp_[a-zA-Z0-9]{36})" " $FILE " 2>/dev/null ; then echo "GUARD BLOCK: Potential secret detected in $FILE " echo "Deployment halted. Remove secret before proceeding." exit 1 fi done Das ist kein ausgeklügeltes KI-Modell. Das ist ein Shell-Skript, das seit Monaten zuverlässig seinen Job macht. 176 Regeln und wie

I had 87% coverage, and we still broke the billing flow on launch day. Not because of a gap in the percentage. Because 87% was covering the wrong things. The tests were written to pass a gate, not to catch a failure. That is a more common story than most teams admit. And the reason it keeps happening is not that engineers are careless. It is that the incentive structure you created made it the rational outcome. The series checkpoint The first three articles in this series built the investment case for testing and then dismantled the received wisdom about how to execute it. We've made the economic argument for automation. We've restructured when quality checks happen across the SDLC. We've replaced the pyramid model with something shaped by risk rather than by code hierarchy. Now, when someone asks: how do you know if it is working? The answer most teams give is their coverage percentage. This article is about why that answer is structurally broken, and why fixing it is a management decision before it is a tooling decision. What coverage percentage actually measures Coverage percentage tracks which lines of your code were executed during a test run. If a line ran, it counts as covered. That is the complete definition. It does not measure whether the test asserted anything meaningful about that line. It does not measure whether both branches of a conditional were exercised. It does not measure whether the specific inputs that cause failures were ever tried. A test that calls a payment function and checks assert response is not None covers the same lines as a test that validates the transaction ID, amount, currency, error code, and retry behaviour. The coverage tool treats them identically. The research on this is unambiguous. A 2017 study by Kochhar et al. examined the correlation between code coverage and actual bug rates across 100 large open-source Java projects. The finding: the coverage of existing test suites has an insignificant correlation with the number of b

We run an automated publishing pipeline entirely on GitHub Actions cron schedules — no server, no queue, just workflows that wake up, do one thing, and commit the result. It mostly works. But there is one behaviour of scheduled workflows that the docs mention in a single quiet sentence and that will silently halve your job frequency if you design around the cron expression instead of around reality: Scheduled workflows do not fire as often as you declare. What we measured We had a feedback-watcher workflow declared at four runs per hour: on : schedule : - cron : ' 7,22,37,52 * * * *' Measured over days, it actually fired one to two times per hour — not four, and not at the declared minutes. Roughly hourly on most days, at inconsistent offsets from the declared slots. We later redeclared it at two runs per hour ( 7,37 * * * * ) — measured result: still one to two runs per hour. The declared frequency changed by 2x; the delivered frequency barely moved. This is not an outage and not a misconfiguration. GitHub's own documentation says the schedule event can be delayed during periods of high load , and that high load times include the start of every hour — which is precisely where naive cron expressions cluster — and adds: "If the load is sufficiently high enough, some queued jobs may be dropped." What the docs understate is the magnitude: in our observation, on a private repo, "delayed" in practice meant "throttled to a fraction of the declared rate, indefinitely." What this breaks The failure mode is subtle because nothing goes red. Every run that happens succeeds. The runs that don't happen leave no trace — no log, no failure email, nothing. You only notice if something downstream depends on the frequency: We had promised a "reply within 15 minutes" SLA on incoming feedback, initially backed by the 4x/hour schedule. The schedule couldn't hold it, so for a while we ran a local 15-minute scheduler as the primary path and kept the workflow as fallback. When we later rel

Classic Machine Learning Through the Eyes of an SRE — Part 7 Picture a client health metric that has been flat at 2 out of 10 for six months. Ask PCA to compress your client-health data and that metric will contribute almost nothing to the directions PCA decides to keep. Not because PCA is broken. Because PCA treats variance as importance, and a signal that barely moves contributes almost no variance. Reduce the data far enough and the independent information it carried is simply not there anymore. But a CSAT frozen at 2/10 is not noise. It is a crisis nobody is escalating. And after compression, it may no longer be available to anything downstream. That is the bet, and in ops data it is frequently wrong. The critical signals are often the quiet ones. There is a cheaper version of the same failure that catches most people first. PCA measures variance in whatever units your features happen to be in, so a metric ranging from 0 to 10,000 can dominate one ranging from 1 to 5 purely because it is bigger. Standardize before you compress, or your first principal component may just be an elaborate way of saying "ticket count." Same class of bug as unscaled features in K-Means and SVM, and it fails just as quietly. What PCA actually is Third answer-finding strategy in the unsupervised set, using the same shorthand as the last two articles. K-Means SEARCHES: iterate and hope. DBSCAN DEFINES: declare a rule and traverse. PCA SOLVES: an eigendecomposition or SVD gives a direct solution rather than an iterative local search. No convergence to babysit, no restarts, no local optima to escape. Two caveats on the word "direct," both worth knowing. Many libraries will use randomized SVD on large matrices, which is approximate and stochastic. And even with an exact solver, eigenvectors are only defined up to sign, so a component can come back inverted between runs or across implementations. The variance explained is identical either way, which is precisely why nobody notices. Hold ont

LangSmith is excellent for debugging live AI systems. But keeping every trace in its extended-retention tier can turn observability into a surprisingly large line item. Today we merged a new archive workflow into langsmith-cli that changes that tradeoff: keep LangSmith for live debugging, continuously archive verified traces to organization-owned private S3, and query the retained Parquet directly with DuckDB. In other words, you can preserve your complete trace history without placing every trace on LangSmith's extended-retention tier. The cost-overrun risk LangSmith currently documents two trace-retention tiers: Tier Retention Published trace price Base 14 days 0.05¢ Extended 400 days 0.50¢ total The 0.45¢ extended-retention upgrade makes an extended trace cost 10× as much as a base trace. That difference becomes material at production volume: Monthly traces Base, 14 days Extended, 400 days Added retention cost 100,000 $50 $500 $450 1,000,000 $500 $5,000 $4,500 10,000,000 $5,000 $50,000 $45,000 These examples use the published per-trace rates before free allowances, plan terms, negotiated pricing, or taxes. Always check the official LangSmith usage and billing documentation before making budget decisions. There is another subtle risk: online evaluators and automation rules can upgrade matching traces when retention extension is enabled. A rule that matches one run upgrades the whole trace, and a thread-level rule can upgrade every trace in that thread. LangSmith currently enables retention extension by default for new online evaluators and automation rules, although you can opt out. At scale, an innocent-looking evaluator or rule can therefore create a much larger bill than expected. The new langsmith-cli archive workflow The new workflow separates live observability from long-term retention: LangSmith live traces (14 days) │ ├── D+2 primary export ───────┐ └── D+12 reconciliation ──────┤ deduplicate by run ID ▼ private S3 / Parquet │ ▼ runs ... --archive (DuckDB)

Every one of my publishing lanes went dark for four days, and every script involved exited with status 0. Nothing had crashed. The files themselves had quietly stopped existing on disk — macOS had uploaded them to iCloud and deleted the local copies to "optimize storage." Why This Matters What it means for automation to depend on its environment When you run 160+ launchd jobs around the clock, the execution environment itself becomes a failure source before your script logic does. Ports get exhausted, processes orphan and pile up, memory never frees — I wrote about that class of resource leak last time. This is a completely different kind of total failure that happened the very next day. The files had become fatal to read . Not a bug in my code. Not a filesystem bug. An unintended side effect of a mechanism macOS runs under the name "optimization." What optimize-storage actually does macOS's "Optimize Storage" (System Settings → General → Storage → Optimize Storage), on a machine with iCloud Drive enabled, uploads files under Desktop and Documents to iCloud and deletes the local copies when free disk space gets tight . In Finder they still look like normal icons, but there is no local data — they are in a "dataless" state. Click one and it downloads automatically. For a human user, that's an acceptable tradeoff. The problem is automation scripts. python3 's open() , pathlib.Path.read_text() , cat , jq , cp — all of them die instantly on a dataless file with Errno 11: EDEADLK: Resource deadlock avoided . The name "Resource deadlock" makes you suspect a deadlock, but this is a POSIX errno code that macOS repurposes to mean "waiting for a file download." No lock is contended. No thread is stuck. The mere fact that "the data isn't local" surfaces to the process as a fatal error code. You can also get EAGAIN (resource temporarily unavailable). That one shows up as a race right after a download starts. The actual damage: four days of zero posts On August 6, 2026, note's a

I Gave Five Graph Databases 256MB of RAM Each. Here's What Broke. CognoDB Cloud's free tier gives you a graph database instance with half a CPU core and 256MB of RAM. That's not a lot. It's also, honestly, a pretty realistic starting point a lot of real side projects and early-stage products live exactly there, on whatever the free tier happens to give them, and find out the hard way what their database does under pressure. So I decided to actually find out. I took CognoDB and lined it up against four other graph databases Neo4j AuraDB, FalkorDB, and ArangoDB gave every single one of them the same tiny resource budget, threw the same 198,050-edge dataset at all of them, and ran the same queries. No cherry-picking, no "best case" numbers. Just: here's a small VM's worth of resources, go. One of the databases I originally planned to include never even made it into the results. It crashed on startup. Not "slow to start" a full segfault, reproducibly, across two different versions, with nothing I threw at it fixing it. More on that below, because it's honestly one of the more interesting parts of this whole thing. The setup, quickly Five candidates going in: CognoDB (mandatory, since that's the actual point of this), Neo4j AuraDB Free, Memgraph, FalkorDB, and ArangoDB. Same dataset for all of them a real social-graph-shaped dataset from Stanford's SNAP collection, ~18.7k nodes and ~198k edges, sized specifically to fit inside every platform's free tier without anyone getting an unfair advantage. Same queries too: I wrote every single query 1-hop, 2-hop, 3-hop traversals, point lookups, filtered lookups, aggregations exactly once, then translated each one into whatever query language a given platform actually speaks. No platform ever got a "friendlier" version of a query than another. And everyone ran under the same 0.5 vCPU / 256MB RAM ceiling, whether that was their real cloud free tier or a Docker container I capped by hand to match. The one that didn't survive Memgra

You said you'd remove that flag after launch. You lied. It's been six months and the flag is still in appsettings.json , the if statement is still in your controller, and nobody remembers which state is "on." This is how codebases turn into haunted houses. Why Cleanup Matters Dead feature flags are technical debt with teeth . They add branches to your code that nobody tests. They confuse new developers who don't know the history. They inflate configuration files and make deployments harder to reason about. And they compound. Every flag you don't clean up makes the next cleanup harder because the cognitive load of understanding the system keeps increasing. The cost of removing a flag is lowest immediately after the feature ships, while everyone still remembers what the thing does. Six months later? Good luck. Track Every Flag You can't clean up what you can't find. Maintain a registry of every active feature flag with: Name Purpose Owner Date created Expected removal date This can be a spreadsheet, an issue tracker, internal documentation, or a dedicated feature flag management system. The format doesn't matter nearly as much as the habit. When you add a flag, add it to the registry. When you remove a flag, remove it from the registry. If your registry contains flags with no owner or no removal date, congratulations: you've found your next cleanup project. Set Expiry Dates Every flag should have a planned removal date when it's created. For example: Release toggles: Remove shortly after the feature ships. Two weeks is a reasonable default. Experiment toggles: Remove when the experiment concludes. Ops toggles: May be permanent by design. Permission toggles: May also be permanent, but document that explicitly. If a flag has been alive longer than its planned expiry and nobody deliberately extended it, it's already a zombie. Treat it accordingly. Make Cleanup Part of the Process Flag cleanup doesn't happen unless someone owns it. Add a cleanup step to your feature compl

I've been running Claude Code as a daily driver for about three months now. It writes Ansible I'd have taken a week to write. It reads a codebase faster than I do. It is, genuinely, very good. It also once wanted to force-push to main , and it wanted to for an extremely good reason. Sit with that for a second, because it's the whole post. The rebase was stuck. Force-pushing would have unstuck it. Every link in that chain of reasoning is sound. The agent wasn't being careless, wasn't hallucinating, wasn't "drifting" or whatever we're calling it this month. It made a locally correct decision with a non-local consequence, which is the exact category of mistake that human code review is worst at catching — because the diff looks fine . It could see the command. It could not see the crater. The thing I stopped doing For a while my answer was to read everything. Every diff, every command, eyes on the screen, hand hovering over Ctrl-C like a man watching a toddler near a staircase. This does not scale, and the reason it doesn't is embarrassing when you say it out loud: reviewing output scales with how much the agent writes. That number is going exactly one direction, and it isn't down. So I flipped it. Instead of reviewing what it produces, I started writing down what it must never do. And here's the good news that took me way too long to notice: that list is short . Not "short for a security policy" short. Short like you can fit it on a napkin. Here's mine: A credential it read an hour ago gets inlined into a source file. A rebase gets stuck, and the fastest route to a green terminal is git push --force origin main . rm -rf "$BUILD_DIR/" runs on the one machine where BUILD_DIR never got set. A version bump gets typed straight into package-lock.json , because that's the file the version number is visibly in. A failing test quietly grows a .skip and CI goes green. Someone runs cat .env "just to see which variables exist." That last one is my favourite, and I'll come back to

Today I shipped contenox 1.0.0. Not by pushing a tag on top of a thousand commits, but as a single commit into an empty repository: the whole tree, one signed tag, binaries built from that tag by CI. The 957 commits that got me there are still public, in the old repository, as history. They are no longer how the project is published. This post is about why, and about what went wrong before I had finished reading the result back. What a commit used to mean GitHub's workflow rests on four assumptions so old that nobody states them any more: A commit is a unit of human intent. Someone decided something and typed it. A pull request is a unit of review. A human reads the diff, because a human wrote it. History is provenance. Who changed what, when, and — through the message — why. Timestamps are labor. The contribution graph on your profile is a diary. All four were true in 2008. For a tree that agents write, none of them survive contact. What my repository actually looked like Some numbers from a tree you can inspect yourself: 957 commits in just over a year, most of them named Checkpoint , Fix tests , Snapshot WiP . Dozens on a busy day. The production Go grew from 17,267 hand-written lines to 134,040 agent-assisted ones. Measured, not estimated. The median file stayed the same size; the number of files and packages did not. At one point 530 uncommitted paths sat in a single working tree. Inside that blob, the file that carried the repository's own conventions had been deleted. Nobody noticed for days, because nobody reviews a 530-file diff. A commit stream like that is not history. It is a log. Reading it tells you nothing about what a human decided — the decisions happened in prompts, in agent declarations, in a policy file — and it tells you one thing with great precision: when the work happened. If you also do client work, a public commit stream is a timesheet you never agreed to publish. Review had quietly inverted, too. I was no longer reviewing commits. I was re

Did you know there's a new major version in town for Puppet Core? You might have heard about it through the grapevine or in the Are You Ready for Puppet 9? webinar that @gpatton and I recently hosted. The wait is over and Puppet Core 9.0.0 is now available alongside Puppet Core 8.21.0. Puppet Core 9 introduces significant runtime and platform changes, moving to Ruby 4.0, OpenSSL 3.5, and other changes, but the essential Puppet under the hood is largely unchanged from Puppet 8. The majority of upgrade effort will center on Ruby 4 compatibility and runtime dependency changes rather than Puppet language changes. If you are staying on the Puppet Core 8.x release track, the latest Puppet Core 8.21 delivers the basic support fixes and security improvements you might need without the major dependency changes found in Puppet Core 9. What matters most for the admins Before upgrading to Puppet Core 9: Test custom facts, functions, types, and providers against Ruby 4.0. Validate any Forge modules you use for Ruby 4 compatibility. Review integrations that depend on OpenSSL behavior. Verify any workflows that still rely on SHA-1. Confirm managed nodes are running supported operating systems. Review any custom code that depends on PSON or multi_json . Check deferred function behavior if you have custom types or providers. Perforce will be rolling out updates to Puppetlabs modules on the Forge based on their priority tier and dependencies. The first batch of these should be rolling out soon. Puppet Core 9.0 highlights These are a few highlights I pulled from the release notes. Make sure to reference the full 9.0 release notes to get all the details about what has changed! Ruby updated to 4.0.5: With a new Ruby baseline some deprecated syntax from older Ruby versions will no longer be compatible. This is the primary focus area for upgrades as you will want to validate your custom code and modules. The latest PDK 3.8.0 introduced some Ruby 4 validators to help you update your syntax

James Arthur shares why sync is the next frontier in frontend architecture. He explains how extending reactivity to the server with Electric and TanStack DB replaces imperative fetching with declarative data bindings. Learn how query-driven sync and local optimistic updates enable engineering leaders to build insanely fast, collaborative, and agentic applications using their existing stack. By James Arthur