标签：#DevOps

⚡ TL;DR Four things about controller mechanics are widely half-understood: what Reconcile receives, where its work comes from, what a periodic resync is, and what a predicate turns off. I built an operator, broke it five ways, and measured each mechanism directly. The reconcile function runs in 2.71ms mean, 77/77 under 25ms , a short resync period costs zero additional API requests , and GenerationChangedPredicate cut steady-state reconciles by 48.5% without touching live repair at all. That last combination is the one that matters at scale. Repo, raw data, and harness: kirPoNik/k8s-drift-operator . 🧩 The four barriers Everyone who runs Kubernetes knows the platform repairs itself. Delete a pod, it comes back. Scale a Deployment by accident, something puts it back. Almost nobody who relies on that property can say how it works, and the gaps are specific and consequential. I keep meeting the same four: People think a controller is told what changed. It is not, and the reason it is not is the single most important design decision in Kubernetes. People think a controller polls the API server. It does not, and knowing what it does instead tells you where your API load actually comes from. People think a resync is a re-check against the cluster. It is not, which is why a short resync period is nearly free — and why the number that is expensive sits somewhere else entirely. People treat a predicate as a pure optimisation. It is a filter with a silent cost, and the cost is not the one the documentation warns you about first. So I built the smallest system that has the self-healing property, broke it on purpose ten times per failure mode, and instrumented each of those four mechanisms until I could state what it does rather than what it is said to do. What I built. One CRD called Echo , holding an image, a replica count, and a greeting. A controller keeps three child objects in sync with it — a Deployment, a Service, and a ConfigMap holding the greeting — with owner referen

I was testing out CH-Ops - an admin GUI for self-hosted ClickHouse - on a simple setup: 1 shard, 2 replicas. Stumbled onto the replication queue almost by accident. Here's what I did: I stopped one of the nodes (let's call it Node B), then inserted some data through the other one (Node A). Just wanted to see what would happen. Then, while Node B was still down, I checked it in CH-Ops. It had stuff sitting in its replication queue. My first assumption was: okay, this must be showing what's left to replicate across the cluster - the total pending replication work. So I switched over and checked Node A, the one that was actually up and had just received the insert. Its queue was empty. That didn't match what I expected at all. If the queue was a cluster-wide "here's what still needs to replicate" view, Node A should've shown something too - it was the one that had the fresh data now waiting to reach Node B. Instead it was Node B, the down one, sitting there with pending tasks. That mismatch is what sent me digging. Turns out the queue isn't cluster-wide at all - it's specific to each ClickHouse instance. Once I brought Node B back up, its queue drained in seconds and the data showed up. That whole experiment is basically the entire post in miniature. Here's the mental model I ended up with. A Queue Belongs to a Replica, Not to the Table This is the first thing to get straight. With a ReplicatedMergeTree table, you can have multiple replicas holding copies of the same data. It's tempting to think of replication as one shared pipe between them. It isn't. Each replica keeps its own local replication queue . So if you see: Replica 1 → queue_size = 0 Replica 2 → queue_size = 25 that doesn't mean 25 operations are waiting somewhere in the middle for both replicas to pick up. It means Replica 2, specifically, has 25 tasks it hasn't finished yet. Once that clicked for me, the rest of the system made a lot more sense. So Where Do These Tasks Come From? Replication in ClickHouse

Every guide I write ends in the same handful of questions. How much hardware do I actually need? What happens when one box dies? Are my backups real or just a feeling? A guide can walk you through a setup, but it can't do arithmetic about your lab — so I built eleven small tools that can, at peira.dev/tools . They're free, none ask who you are, and seven of the eleven keep working once the page has loaded — network unplugged, laptop in a cupboard, whatever. They share one lab profile This is the part that makes them a set rather than eleven unrelated pages. Describe your lab once — tick your services in the sizing calculator, press Save to profile — and the others pick it up. The failure simulator opens with your nodes already modelled; the backup planner knows what data you have; the power-loss playbook knows what's plugged in. Lab doc hands the whole thing back as a Markdown file. Nothing about that profile leaves your browser. No account, no sync, no server that could leak it — which is also why it doesn't follow you between devices. The Markdown export is how you carry it elsewhere. Plan the build Sizing calculator — asks what you want to run and recommends nodes, RAM, and storage. It cares most about RAM, because that's the constraint that actually bites; vCPUs overcommit happily, memory doesn't. Tick "survive one node failure" and it insists on three nodes (a two-node cluster loses quorum the moment one dies). Node failure simulator — kill a node and see which workloads fit on the survivors. It places the critical ones first and names the stranded ones. 3-2-1 backup planner — three copies, two devices, one offsite (the rule CISA recommends ). It's blunt: a snapshot on the same disk as the original is versioning, not a backup. Fix what's broken The overlay network diagnostic is a decision tree born from a miserable afternoon: a container couldn't reach a machine across a Tailscale subnet router, and three layers had to be right — the route in the guest, the ACL

Tuning guides talk about throughput. Nobody pages you about throughput. They page you about symptoms, and the useful skill is mapping a symptom back to a cause before you spend money on hardware. Three failure modes account for most of what I find on inherited servers. Each has a distinct signature. The 502 nobody can reproduce Server has 8GB. PHP-FPM is set to 100 workers. Each worker uses 60MB under load. That's 6GB of PHP, plus MariaDB, plus Nginx, plus the OS. Under normal traffic you never approach 100 workers, so it looks fine for months. Then a marketing email goes out, concurrency spikes, and the kernel runs out of memory. The OOM killer picks a process and terminates it, usually the biggest one, which is a PHP-FPM worker holding an in-flight request. User gets a 502. The application log has nothing, because the process died before it could write anything. Nginx logs recv() failed (104: Connection reset by peer) . Ten minutes later everything looks normal. sudo dmesg -T | grep -i "killed process" sudo journalctl -k | grep -i oom Hits there mean you don't have a mystery. You have a pm.max_children value nobody checked against real memory. The site that degrades all day and resets overnight TTFB is 180ms at 8am. By 4pm it's 900ms. Nobody deployed. Overnight it's fast again because something restarted PHP-FPM. That's OPcache running out of room. When the cache fills, it stops caching new scripts or wipes and rebuilds, and every miss pays full parse-and-compile again. It degrades gradually, which is why it goes unnoticed for months. The counters are oom_restarts and hash_restarts from opcache_get_status() . Here's the part that trips people up. OPcache state is per SAPI. Run that function from the CLI and you're reading the CLI cache, which is empty, separate, and tells you nothing about your site. You have to ask through PHP-FPM. <?php // drop in webroot, lock to your IP, delete when done $allowed = [ '203.0.113.42' ]; if ( ! in_array ( $_SERVER [ 'REMOTE_ADDR'

CERN engineers announced a shift from Red Hat-based distributions to Debian for its accelerator control systems. This decision stems from Red Hat's tightening compiler mandates, which threatened legacy hardware. The transition, focused on 2,200 specialized control machines, is set for completion in late 2026, while CERN's other systems will remain with Red Hat and AlmaLinux. By Olimpiu Pop

> Your load balancer can distribute traffic perfectly and still overload a server. Here's the part of Round Robin we often overlook. Three servers. Six requests. Request 1 → Server A Request 2 → Server B Request 3 → Server C Request 4 → Server A Request 5 → Server B Request 6 → Server C Perfect. Every server got exactly two requests. So the load is balanced... right? Not necessarily. This is where a simple load-balancing diagram can hide a surprisingly important production problem: Equal traffic does not mean equal work. The Problem Isn't the Algorithm Round Robin is beautifully simple. You have three servers: A → B → C → A → B → C Each new request goes to the next server. For many systems, that's perfectly reasonable. The interesting part is what happens when the requests aren't equal. Imagine this traffic: GET /health POST /generate-report GET /profile POST /export-large-file GET /products POST /process-video Round Robin might still produce: Server A → 2 requests Server B → 2 requests Server C → 2 requests On paper: A = B = C In production: Server A ███░░░░░░░ 25% Server B █████░░░░░ 48% Server C █████████░ 91% Same request count. Very different workload. One Request Is Not One Unit of Work A health-check request might finish in a few milliseconds. Generating a large report could involve: multiple database queries significant memory CPU-heavy processing external API calls several seconds of execution To a basic Round Robin strategy, both are still: 1 request And that's the trap. We often think we're distributing load . What we're actually distributing is requests . Those are not always the same thing. Servers Aren't Always Equal Either There's another assumption hiding here. Imagine: Server A → 8 CPU / 16 GB Server B → 8 CPU / 16 GB Server C → 2 CPU / 4 GB Sending roughly 33% of traffic to each server probably isn't what you want. That's where Weighted Round Robin helps. A → Weight 4 B → Weight 4 C → Weight 1 The stronger servers receive more traffic. Better. But

Everyone backs up. Almost nobody restores. So the backup sits there, growing, quietly reassuring, and completely untested. It is not a safety net. It is a photograph of one. The day you need it is the worst possible day to discover that the job has been failing since March. That the archive is encrypted with a key that lived on the machine you are trying to recover. That it holds the database but not the uploads. That it takes nine hours, and the business gave you two. None of that is exotic. All of it is ordinary. An attacker who reaches your data will reach your backups next, because they sit on the same network, under the same account, behind the same key. That is not a backup. That is a second copy of the same hostage. So test the restore. Not the theory. The restore. Into a clean place. With a clock running. By someone who was not there when it was built. Write down how long it took, because that number is your real promise to everyone downstream. Everything else is marketing. Keep one copy somewhere your production credentials cannot reach. Keep one copy that cannot be deleted, even by you, even when you are certain. And do not trust the log line that says the job succeeded. A green tick is a claim. It is not evidence. Security is not only keeping people out. It is being able to come back after somebody gets in. Anybody can copy data. The skill is putting it back while the phone is ringing and nobody agrees on what happened. Practise the boring version too. Not only the fire. One deleted table on an ordinary Tuesday, because that is usually how it starts. Not an attacker. A person, a missing clause, and a bad afternoon. Restore it once before you need it. Then it is a backup. Until then it is a wish with a filename. – Serguey Asael Shinder

You write instructions for your coding agent. It ignores one of them. You rewrite it more forcefully, in bold, with "IMPORTANT" in front. It still ignores it. Before blaming the model, check whether it ever saw the text. Each of the three cases below is documented behaviour of a tool you already use, each one drops part of your instructions on the floor, and none of them prints a warning. 1. Cursor ignores .md files in .cursor/rules Project rules in Cursor must use the .mdc extension. Cursor's own docs put it plainly: a plain .md file there is ignored by the rules system, because it has nowhere to declare the description , globs and alwaysApply frontmatter that tells Cursor when to apply it. So a file sitting in exactly the right directory, with exactly the right content, does nothing. No error at startup, no "rule skipped" line, nothing in the UI. Ten-second check: find .cursor/rules -name '*.md' 2>/dev/null Any output is a rule that isn't loading. Rename to .mdc and add the frontmatter. A detail that makes this worse: people who set up .md rules a while ago report that they used to work. If that's right, a working setup stopped working at some point during an update, and nothing announced it — so "I checked this once" is not protection. 2. Codex truncates your AGENTS.md files — as a set, not one by one Codex reads the AGENTS.md files that apply to your working directory: a global one, the repo root, and the nested ones on the path. It concatenates them, and the 32 KB truncation applies to that combined payload . This is the part that catches people, because every individual file looks fine: AGENTS.md 12 KB ✓ fine packages/api/AGENTS.md 12 KB ✓ fine packages/web/AGENTS.md 12 KB ✓ fine ----- 36 KB ✗ 4 KB never reaches the model Nobody wrote a "too big" file. The rule you carefully put at the bottom of the last one simply isn't there when the model reads. Check it: find . -name AGENTS.md -not -path '*/node_modules/*' | xargs wc -c Add your global ~/.codex/AGENTS.md t

The cloud bill is the only number in most companies that nobody on the team owns until it's already a problem. Engineering owns latency. It owns error rates, p99, uptime, the whole observability wall. Finance owns the invoice. And between those two ownerships there's a gap wide enough to drive a fifth to a third of your cloud spend straight into a wall — which, across the industry, is roughly what happens. The fix isn't a smarter spreadsheet at month-end. Real cloud cost management isn't an accounting function at all — the fix is to stop treating the bill as accounting and start treating it as a product metric: cost per request, cost per tenant, cost per feature, sitting on the same dashboard as latency and error rate, owned by the same people who own those numbers. That's the whole argument. The rest of this post is why it's true and how it's done. The bill is a lagging accounting artifact, and that's the bug Here's how cloud cost is treated almost everywhere. A bill arrives. Someone in finance reconciles it against a budget. If it's higher than expected, a thread gets opened, an engineer gets pulled in, and everyone spends a week spelunking through Cost Explorer trying to reconstruct why a number that's already been spent is what it is. Then it happens again next month. Every part of that loop is broken. The signal arrives weeks after the decision that caused it. The person reading the signal can't act on it. The person who can act on it never sees it. And the unit of measurement — total dollars — tells you nothing about whether the spend was good . A bill that doubled because you doubled revenue is a triumph. A bill that doubled because someone left a debug log streaming to an expensive tier is a fire. Total dollars can't tell those two apart. They look identical on the invoice. This is the same mistake we'd never make with any other production signal. Nobody reviews latency once a month from a PDF. Nobody waits for finance to tell engineering that p99 regressed.

Most S3 bills are wrong, and the fix takes an afternoon. The data sits in the most expensive class AWS offers (S3 Standard, $0.023/GB-month), nobody set a lifecycle policy, incomplete multipart uploads are silently billing for storage you can't even see in the console, and every byte your EC2 fleet pulls from S3 is routed out through a NAT Gateway when a free VPC Gateway Endpoint would do the same job for $0. None of this needs an architecture rewrite. It needs a checklist run in the right order. Here is the order. The savings depend entirely on your access pattern — I will not promise you a number I can't see — but the mistakes below are so common that the question is usually how much , not whether . One number is just arithmetic: cold data that moves from S3 Standard ($0.023/GB-month) to Glacier Deep Archive ($0.00099/GB-month) drops about 96% on the storage line for those bytes, and on an observability platform I ran — logs aged past 90 days into Deep Archive — that is exactly the lever that did the work. S3 cost optimization is the same boring discipline as the rest of the bill: see it, then decide what each byte should actually cost. First, see the bill before you touch it You cannot optimize what you cannot measure, and S3's default billing view tells you nearly nothing useful. Turn on S3 Storage Lens before anything else. The free tier gives you 62 metrics at the bucket level with 14 days of history, and crucially it includes cost-optimization metrics out of the box — including "Incomplete multipart upload bytes greater than 7 days old," which is the single most common source of money disappearing into storage nobody knows exists ( AWS S3 Storage Lens docs , accessed 2026-06-18). Storage Lens free metrics answer the three questions that decide everything that follows: Which buckets hold the most bytes? What storage class is that data sitting in right now? Where are the incomplete multipart uploads? For deeper per-prefix analysis or a longer history, Advanced

If your AWS bill has a NAT Gateway line, you are paying twice for the same packet: once for the gateway to merely exist, and again for every gigabyte it carries. The fix for most teams is dull and free. Add an S3 and a DynamoDB gateway endpoint, route the heavy traffic away from NAT, and only then argue about anything fancier. That single change is free to turn on, takes minutes, and stops the most expensive traffic from ever touching the meter. This is a playbook, not a lecture. The trick with NAT Gateway pricing is that the two charges hide in different places on the bill, so most teams only ever see half of it. Numbers first, then the fixes, in the order I would actually do them. What you are actually being charged for NAT Gateway has two charges, and people forget the second one until they read the bill closely. Hourly charge — you pay for every hour the gateway is provisioned and available, whether or not a single byte moves through it. In us-east-1 (N. Virginia) and us-east-2 (Ohio) this is $0.045 per NAT Gateway-hour . That is roughly $32.85 a month per gateway just to keep the lights on. Partial hours bill as full hours. Data processing charge — you pay $0.045 per GB processed through the gateway, in the same region, on top of the hourly charge. This applies to every gigabyte, inbound or outbound, regardless of source or destination. And then there is the part the pricing page mentions almost in passing: standard AWS data transfer charges still apply on top. NAT processing is an extra meter on traffic you were already paying to move. The hourly charge is fixed and visible. The per-GB charge is the one that catches teams out, because it scales with traffic you mostly cannot see: package installs, container image pulls, S3 reads from private subnets, telemetry shipped out, cross-region calls. The rate varies by region (it runs higher in places like São Paulo, where both the hourly and per-GB rates sit around $0.093), so check your own region rather than trusti

Give me read access to a $50,000/month AWS account and I will tell you within a day where the first 20-30% is hiding, because on a mid-size bill it is almost always hiding in the same four places, in the same order: data transfer you can't see in the console, instances sized for a load test that ran two years ago, on-demand pricing on a baseline that never moves, and storage rotting in the most expensive class AWS sells. None of this needs an architecture rewrite. AWS cost optimization, at least the first and biggest pass of it, is just the bill read in the right order by someone who knows where AWS buries the meter. This is the order I work. It is the same audit I run on every account I'm handed, and it is the offer — if you want me to run it on yours, the post ends with how. But you can run most of it yourself today, and you should, because nobody is going to care about your bill as much as you do. A note before the recipe: I deal in ranges, not promises. The exact saving on your account depends on what you've built. What I can promise is that the mistakes below are common enough that the question is usually how much , not whether . Hour zero: get the real bill, not the dashboard Before touching a single resource, I want the granular data. The AWS console's cost dashboard rounds, groups, and hides the things that matter. Two tools give you the truth. Cost Explorer , with rightsizing recommendations turned on, is the fast view — group by service, then by usage type, and the bill stops being one big number and starts being a list of decisions. Resource-level and hourly granularity costs extra ($0.01 per 1,000 usage records per month), but for one audit pass it's worth pennies. The Cost and Usage Report (CUR) is the ground truth — line-item, hourly, every charge AWS makes, delivered to your own S3 bucket. Generating it is free; you pay only the few cents of S3 storage. If you're going to do this seriously, set up CUR (now delivered via AWS Data Exports) on day one. E

I rent six dedicated servers from a company in Germany. Together they have more cores, more memory, and more SSD than most production clusters I worked on a decade ago. I run my own Kubernetes on them. Not managed. Not EKS. Not GKE. The whole stack, from the immutable OS up to the workloads. People who hear this ask me why, and the question usually arrives in one of two tones. The dangerous tone is "that's amazing, how do I do it" . The responsible tone is "why on earth would you do that to yourself" . This post is for the second group. Why the cloud is the right answer for almost everyone Let me get this out of the way honestly: by every conventional metric, I should be using the cloud. Managed Kubernetes has become genuinely good. EKS has dramatically improved over the last three years. GKE has always been better than people gave it credit for. The serverless options are mature. The serverless databases are mature. The observability is mature. The bill is predictable in the way a Tuesday is predictable. Self-hosting violates almost every assumption that makes a startup productive. Time is the most expensive resource you have. The cloud sells you abstractions that turn that time into product. Running your own substrate means the time goes into the substrate. If you are trying to ship a product to customers — go use the cloud. Stop reading this post. It will only confuse you. What the cloud does not sell you Here is what the cloud will not sell you, even if you are willing to pay extra: control over your own roadmap. The cloud's roadmap is the cloud's. They decide which APIs deprecate. They decide which regions get the new feature. They decide what your egress bill looks like. They decide whether your monitoring vendor — sitting on top of their infrastructure — is allowed to charge you eight times what it would cost you to host the same software yourself. They decide whether the small ML company hosting your fine-tuned model gets acquired by someone with very differ

Docker management, monitoring that goes deeper than a green dot, backups I have actually restored, and everything else that showed up once deploying stopped being the hard part. Moving off Vercel solved exactly one problem: deploying. Everything else I used to get for free, quietly, as part of the platform, I now had to go find and wire up myself. A month into running my own server, I had a list of ten tools taped to the inside of my head, each one solving a problem I did not know I had until it happened to me at a bad time. This is that list, in the order I actually needed them, with the mistake or the moment that made me install each one. I lean JS and Rust wherever I can, partly out of preference and partly because those are the tools that keep pace with how fast the rest of my stack moves. A couple of these are not JS or Rust at all, and I kept them anyway because they were simply the best tool for the job. 1. Dokploy, for everything I wrote about yesterday This is the one I already spent an entire post on, so I will keep it short here. Push to main, Dokploy builds the container, Traefik points a domain at it, done. Four apps running on one $24 droplet, and adding a fifth would not move the bill. If you deploy anything with Docker and are still doing it over SSH, start here. Everything else on this list assumes you already have a platform under you, not just a server. 2. Neon, for the database half of preview environments The first crack after Dokploy was previews. Dokploy gives every pull request its own preview URL, which is one of the nicest things about the whole setup, right up until every preview hits the same production database. I corrupted a batch of test data twice before I noticed what was happening. Neon branches Postgres the way git branches code, copy-on-write, so a preview PR gets its own preview database that costs almost nothing until it actually diverges from main. The storage engine underneath is written in Rust, and it quietly closed the othe

Run this against the real, published image and watch it fail: docker run --rm --entrypoint trelix-mcp ghcr.io/sairam0424/trelix:3.2.1 --version Exit code 127. Not a crash inside trelix-mcp, not a stack trace, not a permissions error — 127 is the shell's own way of saying the binary you asked for does not exist. And it didn't. The console script trelix-mcp is supposed to install as part of every trelix package was simply absent from the image, on both the slim tag and the -local tag, for the entire life of the 3.2.1 release. Every unit test in the suite was green. Every line of source that builds trelix-mcp was correct. The thing a user would actually get from docker pull did not have the binary its own --version flag implies exists. This article covers four releases — v3.2.2, v3.2.3, v3.2.4, and v3.2.5 — spanning 173 commits and 88 changed files since v3.2.1, which is where the last article in this series left off. That one was about tests that pass without exercising the code they claim to cover: a MagicMock standing in for a real embedder, an all-ones attention mask that makes masked and unmasked math identical, a unit test that asserted a bug as its own specification. This one, on the heels of the mutation-testing push that closed out that arc, is about a different and in some ways more uncomfortable failure mode: tests that pass while exercising the wrong artifact entirely. A green pytest run against src/ says nothing about whether the wheel on PyPI, the image on GHCR, or the binary on the GitHub Releases page actually does what it claims. Those are three separate build products, built by three separate pipelines, and none of trelix's 4,353 collected unit tests had ever touched any of them directly. v3.2.2 through v3.2.4 is the story of finding that gap and closing it with an actual gate, not a promise to be more careful next time. v3.2.5 is a short postscript proving the discipline stuck. The Docker image that shipped without its own server The 127 above wasn't

Nineteen OOM kills in nine days. Ten WordPress apps on a 32GB shared box. One of them a client site that took a CPU spike on 14 July during a paid campaign burst, and pushed the whole tenant into the wall. This post is the diagnosis before the rebuild. What we actually found when we stopped guessing. Two layers of Cloudflare, one page cache plugin, one preloader being silently challenged, and a language subpath that was cold every time it mattered. I'm writing it partly for anyone who runs multi-tenant WordPress on Cloudways or similar, and partly as a reminder for future-me. There's a checklist at the end. Steal it. The client is anonymized throughout. Every number is real. The stack Traffic hits two Cloudflare layers before it reaches origin. Both are Cloudflare, but they're different zones on different accounts, and they own different things. [ visitor ] ↓ [ Upstream Cloudflare zone (managed by a third party) ] ← DNS, SSL, HTML edge cache ↓ [ Cloudflare Enterprise add-on sold by Cloudways ] ← WAF, bot, rate limit, AI crawler block ↓ [ Cloudways origin: nginx + PHP-FPM ] ↓ [ WordPress + WPML + Elementor + FlyingPress ] Two Cloudflares isn't a mistake. The domain has been on Cloudflare via an upstream party since before the site moved to Cloudways. When Cloudways later offered a Cloudflare Enterprise add-on for its security stack, we kept both. We manage the Cloudways side. We don't own the upstream zone, which shapes what we can and can't do without a request going out. The trap is that both layers can cache HTML, and both can serve security challenges. If nobody writes down which layer does what, they fight. Our ownership split ended up like this: Layer Owns Upstream Cloudflare (third party) DNS, SSL, HTML edge cache, purge lifecycle Cloudways CF Enterprise add-on WAF, bot management, rate limiting, AI crawler blocking, ScrapeShield, Browser Integrity Check FlyingPress Origin page cache, Cloudflare integration pointed at the upstream zone, purge rules Cloudways "

Security Bolted on at the End Is Not DevSecOps The most common failure pattern in teams that claim to "do DevSecOps" looks like this: build the pipeline, ship the feature, and run a security scan right before release — treating security as a checkbox at the end of the process instead of something built into every stage of it. That's not DevSecOps. That's a security review with extra steps. Real DevSecOps means security is woven into the pipeline itself — scanning dependencies on every commit, catching misconfigurations before they're deployed, and treating a vulnerability the same way you'd treat a failing test: something that blocks the pipeline, not something reviewed manually after the fact. This guide is for people who already have some DevOps or backend foundation and want to understand what it actually takes to move into a DevSecOps-focused role — not just "add security" to an existing skillset, but understand the mindset shift that makes the discipline distinct. This post originally appeared on the Ciphemic Academia blog . What "DevSecOps" Actually Requires DevSecOps sits at the intersection of three skill areas, and a real role expects working competence across all three, not deep expertise in just one: DevOps fundamentals — CI/CD pipelines, infrastructure-as-code, containers, the same core skills a cloud/DevOps engineer needs Application security — understanding common vulnerability classes, how to find them, and how to actually fix them, not just recognize their names Security automation — the specific skill of embedding security checks into a pipeline so they run automatically, consistently, on every change That third point is what actually distinguishes DevSecOps from "a DevOps engineer who also cares about security." It's specifically about automation and process — making secure practices the default path, not an extra manual step someone has to remember to do. Step 1: Confirm Your DevOps Foundation Is Solid DevSecOps is not an entry point into DevOps —

I run about thirty scheduled jobs on a single Windows box. Some are scrapers, some generate content, some are trading bots, some just check that the other jobs are alive. Most of them were written and are maintained by an AI coding agent that I let run unattended. Over three months, every one of the failures below reported success . The scheduler said LastTaskResult = 0 . The logs looked fine or didn't exist. And nothing had happened. If you only take one thing from this post: stop checking exit codes, start checking artifacts. I'll get to why at the end. First, the seven ways I got lied to. 1. The wrapper that always returns 0 To stop console windows flashing on my desktop every few minutes, I wrapped each scheduled task in a tiny VBScript launcher: Set WshShell = CreateObject ( "WScript.Shell" ) WshShell . Run "cmd /c "" python job.py >> job.log 2>&1 "" " , 0 , True 0 hides the window. True waits for completion. I assumed True also meant the exit code came back. It does not. WshShell.Run used as a statement discards the return value, so wscript.exe exits 0 no matter what the child did. I found this because a content pipeline had been dead for five days while the scheduler reported green every single day. The fix is to call Run as a function and pass the value out: Set WshShell = CreateObject ( "WScript.Shell" ) exitCode = WshShell . Run ( "cmd /c "" python job.py >> job.log 2>&1 "" " , 0 , True ) WScript . Quit ( exitCode ) Note the parentheses — required when you're taking a return value. After fixing this across 17 launchers, one task showed a non-zero result for the first time in its life . It had been failing for weeks. 2. The last line of your batch file overwrites the exit code Fixed the launcher, still got false greens. The next layer down was a .cmd shim: node pipeline .js >> run .log 2 >& 1 echo [ done ] exit code %errorlevel% >> run .log That echo is the last command, echo always succeeds, so the batch file returns its exit code — zero — regardless of wh

Level 300 Some time ago we created a blog showing the capabilities of AI DLC and SDD to create quick and efficient prototypes as a MVP, the results were amazing, however, we omitted something: DevSecOps best practices and CICD for the workload. The prototype was built with a serverless framework and modern cloud-native application patterns. However, moving from a working MVP to a production-ready solution requires stronger alignment with DevSecOps, CI/CD, and operational excellence. That transition leaves several important questions open: Security posture • What security vulnerabilities exist in the solution? • Is the platform ready to withstand common web attacks? Code quality and production readiness • What is the overall quality of the code? • Is this truly an example of an enterprise-ready solution? Cloud compliance and misconfiguration risk • Which cloud security compliance gaps still need to be addressed? • Are there misconfigurations that could create operational or security risk? The other side of this challenge is organizational readiness. Many companies are enabling development teams with assistants such as Kiro, Claude, Cursor, and similar tools. However, without a mature process to review, scan, govern, and manage code at scale, these tools can introduce high costs, expand security risks, and growing technical debt. For the Agentic AI era, DevSecOps maturity is no longer optional. A secure software development lifecycle, policy-driven development, and zero-trust principles must become core operating requirements rather than afterthoughts. Open loop – Starter point a classical DevSecOps CICD system Suppose that the company already has a continuous integration and delivery framework and tools for that kind of workload with classical tools, in terms of maturity level and L3 Defined and Managed here we have Security gates integrated in CI/CD; SAST/SCA/secrets/IaC; centralized findings; quality gates. The maturity model we use We measure against a five-level

The permission prompt was the last brake on my fleet, and it was in the wrong place. A prompt fires when a human is sitting there to read it. My agents do most of their work when nobody is: the nightly drain, the noon pass, the headless jobs that read the open web. Those run with prompts skipped, by design, because a prompt nobody answers is a stalled job. So the protection was strongest exactly where I was already watching, and absent where the unattended work runs. What replaced it is a hook. The harness runs a small shell script before every tool call, in every session, in every permission mode, bypass and headless included. The script reads the call as JSON and either lets it through or exits with the code that feeds its message back to the model. Until last week it covered one class: the moves an injected instruction would need, reading a credential file, dumping the keychain, piping a download into a shell. It now covers the class I had left to the prompt: force pushes, a hard reset or a branch swap in the one working tree several live sessions share, a recursive delete aimed at a home or project root, a package release. The hook exists because of where the old rules lived. One of my contract rules was written in four documents and enforced in one place: a deny list that loads only for a session rooted in a particular directory. Both sessions that broke the rule were rooted somewhere else, so they met no rule at all, while the doctor that checks the setup went green, because it grepped the deny list's text. A rule enforced one directory wide is enforced in the one place the violation was never going to come from. A hook loads everywhere, so it is where a rule that binds every session has to live. The rule for adding a rule is a throughput rule, not a caution rule. A rule earns its place only if it fires almost never, or if it prevents the kind of cross-session destruction that forces other sessions to redo their work. Anything frequent and recoverable stays ou