标签：#devops

An agent patch is only as trustworthy as the checks it cannot rewrite. If properties, fixtures, and flake policy live in the same tree as src/ , the diff can weaken the proof. Move the oracle out of the writable tree and run it as a control loop with hysteresis, not as a skip list. Co-located tests fail this requirement in a predictable way. The agent adds an assertion that matches the new code. A fixture grows a default that hides a broken parser. A flaky case becomes skip . The suite stays green. Production still drifts. This article proposes a sidecar oracle: human-owned properties, sealed fixtures, and a two-threshold flake freeze. The design is a workflow, not a production case study. Treat the code as a proposed runner you can execute locally, not as a claim about a live fleet. What the loop decides The loop answers three questions on every candidate patch: Do independent properties still hold on generated inputs? Did the patch mutate a sealed fixture or depend on an unsealed one? Is a failing test a regression, or does it belong in a measured freeze? A skip list answers none of those. It only records that someone got tired of a red job. Layout: oracle beside the repo, not inside the diff Keep the application repo writable for the agent. Keep the oracle in a second directory that the agent cannot include in its patch. app/ # agent may write src/, not oracle paths src/ pyproject.toml oracle/ # human-owned; hashed before every gate properties/ test_invariants.py fixtures/ manifest.json http_empty_body.json flake_ledger.json path_deny.txt run_gate.py path_deny.txt is the first control, not the last. If the patch touches oracle files, tests the agent authored, or lockfiles it did not need, the gate fails before pytest starts. # oracle/path_deny.txt oracle/ **/test_*.py **/*_test.py **/conftest.py **/__snapshots__/ The deny list is deliberately blunt. Agent-authored tests can still exist as scratch. They do not count as evidence. Step 1 — Hash the oracle before the

Ever Wondered How Netflix Never Seems to Go Down? Think about this for a second. Netflix has over 260 million subscribers worldwide. People are watching shows in Tokyo, London, Lagos, and New York — all at the same time. And yet, when was the last time Netflix crashed on you? Now think about your favourite food delivery app. You open it, order food, track your driver in real time, and get a notification the moment your burger arrives. All of that happens in seconds. Behind all of this is a way of working called DevOps. And by the end of this article, you'll understand exactly what it is — no jargon, no complicated diagrams, just plain English. The Old Way (And Why It Was a Nightmare) To understand DevOps, we first need to understand the problem it solved. Imagine a software company in the early 2000s. They had two completely separate teams: The Developers — the people who wrote the code and built new features The Operations team — the people who managed the servers and kept everything running These two teams barely talked to each other. Developers would spend months building new features, then hand over a massive pile of code to the operations team and say "here you go, make it work." The operations team would panic. They hadn't been involved in building it, had no idea what it did, and now they had to deploy it to millions of users without breaking anything. The result? Deployments took weeks. Bugs slipped through. Systems crashed. Customers complained. And the two teams blamed each other. Sound stressful? It was. So What is DevOps? DevOps is simply the practice of bringing developers and operations teams together to build, test, and release software faster and more reliably. The name itself is a combination of Dev (Development) and Ops (Operations). Instead of two teams working in silos, they work as one team with shared goals, shared tools, and shared responsibility. Think of it like a restaurant kitchen. In a badly run kitchen, the chefs cook the food and just s

The outage that teaches you deployment A service passes every test locally. It fails in staging because the API calls localhost:5432 for Postgres — but Postgres is in another container, reachable only as db:5432 . This is not a Docker problem. It is a boundary problem: your code assumed an environment it does not own. Engineers who have shipped on AWS Lambda already avoid a class of these mistakes. They never SSH into a function to hot-fix. They inject config at deploy time. They treat each invocation as disposable. Containers reward the same discipline with different vocabulary. This article maps what transfers, what breaks, and what I require before any Python backend goes to production in a container. What serverless already taught you Immutable deployments Lambda versions are replaced, not patched. Container images work the same way: build a new image, roll out, roll back by tag. If your incident runbook includes "edit files inside the running box," you have a design problem. Configuration at runtime Secrets belong in Secrets Manager or injected env vars — not in source control, not in the image layer cache. Docker does not change the rule; it changes where you mount the values. Single responsibility per unit One Lambda, one job. One container, one main process. Compose and Kubernetes add orchestration; they do not remove the rule. Cold start awareness Slim packages on Lambda map to slim base images ( python:3.12-slim , multi-stage builds). Startup time affects autoscaling and health-check windows the same way cold starts affect user-facing latency. If you understand why a Lambda deployment package should stay small, you understand why a 2 GB container image is a liability. Where the mental model breaks 1. Network identity Inside Compose or Kubernetes, localhost is the container itself. Services discover each other by DNS name ( api , db , redis ). This is the most common first-production failure I see in teams moving from bare metal or single-host deploys. 2. P

1. Provisionar não é configurar No artigo anterior desta série, vimos o OpenTofu como uma alternativa (ou substituto direto) ao Terraform para a tarefa de provisionar infraestrutura — criar VMs, redes, bancos de dados gerenciados, buckets. Mas provisionar um servidor é só o primeiro passo: depois que a VM existe, alguém precisa instalar pacotes, configurar usuários, aplicar hardening, subir a aplicação e manter tudo isso consistente ao longo do tempo. É nesse espaço que o Ansible entra — e é comum ver os dois trabalhando juntos no mesmo pipeline, não como concorrentes. 2. Onde o Terraform para e o Ansible começa A distinção mais útil na prática é: Terraform (e OpenTofu) são ferramentas de provisionamento : elas conversam com APIs de nuvem para criar, atualizar ou destruir recursos. O modelo mental é declarativo e orientado a estado desejado do recurso : "quero uma VM com esse tipo de instância, nessa rede, com esse disco". Ansible é uma ferramenta de gerenciamento de configuração : ela conecta em máquinas já existentes (via SSH, sem precisar de agente instalado) e executa tarefas para deixá-las em um estado desejado: "quero o Nginx instalado, essa versão, esse arquivo de configuração, esse serviço rodando". Não é incomum ver os dois no mesmo pipeline: o Terraform cria a VM e expõe o IP como output; o Ansible usa esse IP para conectar e configurar o que está dentro dela. Um cuida do "hardware" (ainda que virtual), o outro do "software". 3. Conceitos fundamentais do Ansible Antes de ver exemplos reais, vale fixar o vocabulário: Inventory: a lista de máquinas que o Ansible gerencia, agrupadas logicamente (por exemplo, webservers , databases ). Pode ser um arquivo estático (INI ou YAML) ou gerado dinamicamente (ex.: a partir de tags de uma conta AWS). Playbook: um arquivo YAML que descreve, em ordem, quais tarefas ( tasks ) devem ser executadas em quais grupos de máquinas do inventory. Module: a unidade de trabalho executada por uma tarefa — existem módulos prontos para

If your AI application suddenly becomes 300–500ms slower after adding an AI gateway, the first...

Security controls in AI systems often end up duplicated across applications. One team adds prompt validation, secrets detection, PII filtering, authentication, logging, and rate limits to an LLM application. A second team builds similar controls around another provider. Once agents enter the mix, the organization has several implementations of policies that should be shared. An AI gateway provides a central place to enforce those controls. Traditional infrastructure already centralizes TLS termination, authentication, rate limiting, traffic routing, and observability instead of rebuilding them in every microservice. The same pattern applies to AI traffic. What is an AI gateway? Without a gateway: +--> OpenAI Application -------+--> Anthropic +--> Gemini With a gateway: Applications | v +-------------------------+ | AI Gateway | | | | Authentication | | Rate limits | | Guardrails | | Routing | | Logging / observability | +------------+------------+ | +------+------+ | | | v v v OpenAI Anthropic Gemini Products in this space include Bifrost , Kong AI Gateway , LiteLLM , and Cloudflare AI Gateway . Their feature sets differ, but each can route model traffic through a common control layer. Security policy can run at that layer before traffic reaches a provider. Why application-level guardrails become painful To prevent users from accidentally sending credentials to an LLM, a simple implementation might look like this: def ask_llm ( prompt ): if contains_secret ( prompt ): raise SecurityError ( " Potential secret detected " ) return llm . chat ( prompt ) For one application, this is reasonable. Across 30 services, enforcement starts to drift. Some services call OpenAI directly, others use Anthropic, and several teams maintain their own wrappers. One application omits the secret check, another uses an old version, and a third checks prompts but no other AI interactions. Policy enforcement now depends on convention. Moving enforcement to the gateway changes the model: Requ

developer context switching security DevSecOps flow state developer velocity security alerts batch security patching ROI cost of context switching developer productivity security security alert fatigue developer cognitive load ad-hoc security patching interrupting developer flow engineering vp productivity metrics DevSecOps velocity context switching recovery time 23 minute recovery context switch batching security alerts SLA-backed fix campaigns security SLA for developers minimizing context switching feature delivery vs security developer experience DevSecOps The Real Cost of Context Switching What Security Alerts Actually Do to Developer Flow Back to blog What interruptions actually cost Is it worse for developers specifically? The research says probably yes The alert volume isn't imaginary — but be careful which numbers you cite The fix: batch the routine work, protect the calendar The important exception: not everything can wait for the batch A more honest way to estimate the ROI The takeaway Sources The Real Cost of Context Switching: What Security Alerts Actually Do to Developer Flow Companies keep investing in better frameworks, tighter deployment gates, and broader platform suites — and feature delivery keeps getting slower anyway. For engineering leaders trying to explain that paradox to the board, the usual suspects (headcount, tooling, talent) rarely hold up. The more useful place to look is something less visible: how often developers get pulled out of what they're doing, and what it costs them to get back in. As "shift-left" security practices spread, developers absorb a steady stream of vulnerability alerts, automated pull-request comments, and one-off Jira tickets throughout the day. The goal — a more secure codebase — is the right one. The delivery mechanism is often the problem. Scattering fixes across random moments in the workday erodes productivity without necessarily making the codebase safer any faster. The alternative a growing number of engi

There was a deployment pipeline at one job that everyone called "Tomasz's script." It did roughly nine critical things in a precise order, it had saved us thousands of hours over the years, and exactly one human on the planet understood how it worked. When Tomasz was around, this was invisible. When Tomasz went on holiday and the script failed at eleven at night, it stopped being a convenience and became the single scariest object in the company. We stood around a terminal reading code none of us had written, afraid to touch it and unable to leave it alone. This is the quiet paradox of automation. The whole point is to remove human effort, and it succeeds so completely that the humans forget how the thing works, or never learn in the first place. A manual process, for all its tedium, keeps knowledge distributed across everyone who performs it. A perfect automation concentrates that knowledge into whoever wrote it and then lets everyone else safely forget. The more indispensable the script becomes, the more dangerous its single point of understanding grows. What makes it worse is that these scripts accrete. They start simple and legible, then someone adds a special case for a weird environment, then a workaround for a vendor bug, then a hack to handle the one customer who is different. Each addition makes sense in the moment and makes the whole slightly more opaque. By the time it is truly load-bearing, it has become a small undocumented system that only its author can reason about, and its author is a busy person who is one job offer away from taking all of it with them. I have stopped treating a working automation as finished. Working is only half the requirement. The other half is that at least one other person can read it, understand what it does, and safely change it. That means the script explains its intent, not just its steps. It means the tribal knowledge lives somewhere other than one skull. It means occasionally, deliberately, having someone who did not wr

WebLLM: The Rise of AI That Runs Directly in Your Browser For the last few years, the dominant architecture for generative AI has been straightforward: Your application → Cloud API → Large Language Model → Response Every time you interact with an AI application, your prompt or data is typically sent to a remote inference service. But a different architecture is emerging: Your browser → Local AI model → Your device's GPU This is where WebLLM becomes interesting. WebLLM is an open-source, high-performance inference engine that allows large language models to run directly inside a web browser using WebGPU . The inference can happen on the user's device rather than on an application server. That seemingly simple change has significant implications for privacy, cost, offline AI, AI agents, enterprise applications, and cybersecurity . What exactly is WebLLM? WebLLM is not another large language model like Llama, Qwen, Gemma, or Mistral. Instead, think of WebLLM as an AI runtime for the browser . It provides the infrastructure required to load compatible open-source models and perform inference using the user's hardware. The basic architecture looks like this: Traditional AI User ↓ Web Application ↓ Backend Server ↓ LLM API / GPU Infrastructure ↓ Response With WebLLM: Web Application ↓ WebLLM ↓ WebGPU ↓ User's GPU / Device ↓ Local LLM inference WebLLM uses WebGPU for hardware acceleration and provides an OpenAI-compatible API, making it possible to integrate local models into JavaScript/TypeScript applications using familiar patterns. Why does this matter? The most important word is: Local Instead of sending every request to a remote AI service, an application can perform inference locally in the browser. That creates several potential advantages. 1. Privacy Consider an employee using an AI-powered security assessment tool. They may upload: Architecture diagrams Security policies Source code Vulnerability reports Compliance evidence Internal documents Configuration files W

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it. Some of the most misleading moments in building software happen when the page looks finished. The button is there. The layout is polished. The flow works in a test account. The code has been merged. It is very easy to look at that and think the product has moved forward. Then production reminds you that a product is larger than its frontend. I learned this repeatedly while building Eterna Clarity. A customer-facing change could depend on application code, a database function, authentication, storage rules, an email template, environment configuration and the way a demo account was isolated from real customer data. If one of those pieces stayed behind, the screenshot could be correct while the product was not. That changed the way I think about releases. A release is not “the code shipped.” A release is the smallest complete set of owned systems that have to advance together for the accepted behavior to become true in production. The browser can hide a lot of unfinished work Frontend work is unusually visible. That makes it easy to use as a proxy for progress. Back-end state is less visible. So are permissions, production configuration, storage policy, transactional email, tenant boundaries and data migrations. They tend to reveal themselves only when something goes wrong. That asymmetry can create a strange kind of false confidence. A team can spend hours polishing the thing a customer sees while the systems underneath it still describe an older product. In Eterna, the correction was to stop treating the repository as the whole release. Source code still matters. It is simply one owner among several. If a new customer flow requires a database change, the production database has to advance. If it requires a new authentication behavior, the production auth configuration has to advance. If it depends on storage permissions, those permissions have to exist in

The AWS API has become infrastructure's common language, and a whole ecosystem has grown up around running it somewhere other than AWS. Some tools mock it for testing. Others implement it for real. Knowing which is which saves you from deploying a dev tool to production or wiring a production platform into your CI pipeline. Two categories The tools split into emulators and real cloud platforms. Emulators intercept AWS API calls and return plausible responses without provisioning real infrastructure, where state is usually ephemeral, VMs never boot, and the goal is behavioural approximation fast enough for a developer's inner loop. Real cloud platforms provision actual infrastructure where EC2 calls boot real virtual machines and block storage carries real persistence guarantees. Emulators Moto Moto ( github.com/getmoto/moto , Apache 2.0, 8,400+ stars) has been around since 2013, making it the oldest option here. It works differently from the rest because rather than running a local server, it patches boto3 calls in-process through a test decorator. A function wrapped in @mock_aws intercepts all AWS SDK calls and returns mock responses without any network traffic. This makes it fast and easy to drop into Python test suites, but it only works for Python. Teams using the AWS CLI, Terraform, or Go SDKs need a server-based option. LocalStack LocalStack ( github.com/localstack/localstack , 64,000+ stars) is the dominant name in local AWS development. It runs as a Docker container exposing the AWS API on localhost:4566 and covers over 120 services. In March 2026, LocalStack archived its Community Edition repository and moved core services behind a paid plan. A free tier remains for non-commercial use and open source projects, but the Base plan covering Cloud Pods persistent state costs $39 per month and the Ultimate plan runs $89 per month. Teams that depended on CE for commercial CI pipelines are now evaluating alternatives. Floci Floci ( floci.dev , github.com/floci-io/f

We’re a small team building Sealshot, a free and open-source screenshot app for macOS. We started working on it because screenshots often become disposable files. We use them for bug reports, QA, documentation, support, and security work, but later they can be hard to find or reuse. They can also accidentally contain sensitive information such as emails, API keys, tokens, internal URLs, or customer data. Sealshot is built around a simple idea: Treat screenshots more like documents than temporary images. It supports: region, window, and scrolling capture screen recording editable annotations OCR and searchable screenshot archives sensitive information detection before sharing encrypted local storage local metadata generation Everything is processed locally on the Mac. It’s open source and free, and we’re still actively improving it. We’d really appreciate feedback, especially from developers, QA engineers, support teams, and people working in security. Website: https://seal-shot.com/ GitHub: https://github.com/ldeng83/Sealshot

Last Tuesday, my free endpoint returned a valid JSON contract. The next call returned a summary. Same prompt. Same model label. No version bump. I almost merged code that expected a schema and instead got a paragraph. Free tiers are not the enemy. Silent drift is. When you wire a free AI server into your PR pipeline, you accept three facts: shared compute, changing model configs, and zero guarantee. So you need gates that fail closed. This is the checklist I now run before any AI-generated suggestion touches a merge branch. I built these gates against an open-source gateway called MonkeyCode. Why? It gives solo devs free model access and a free server for trial workloads. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Convenient, yes. Safe by default? No. So I test every claim. Gate 1: Pin the response contract Your prompt must define an exact shape. For a bug triage task, I require a JSON object with severity , summary , and file fields. If the response is not parseable JSON, the gate fails immediately. { "severity" : "high" , "summary" : "Null pointer on empty input" , "file" : "src/parse.ts" } No fallback. No partial acceptance. Gate 2: Snapshot a baseline Run the same prompt ten times. Record output length, hashes, and tokens per call. Store those as baseline.json . Later, compare every new response against that range. for i in $( seq 1 10 ) ; do curl -s your-monkeycode-endpoint -d '{"prompt":"triage this bug"}' \ | jq -r '.output' | sha256sum done If the hash variance crosses an evidence threshold, the gate flags it. Gate 3: Time-box and cost-cap Free servers queue. You need a timeout and a token budget. I use 8 seconds and a hard cap of 600 tokens. The gate reads usage metadata from the response and rejects when either limit is hit. if response . elapsed > 8 or response . usage . total_tokens > 600 : reject ( " over budget " ) Track this weekly. Drift often starts as a slow climb. Gate 4: Apply semantic checks Gates are not jus

HashiCorp is positioning HCP Terraform as the governance and control plane for a new generation of AI-driven infrastructure, arguing that the rapid adoption of coding agents is shifting the biggest infrastructure challenge from writing configuration to verifying and safely executing it. By Craig Risi

A syndication job noticed before I did A scheduled task publishes one blog post a day to a developer community. It fetches the article from my own site, converts it, and posts it. At 10:00 it failed four times with this: Server error '521 <none>' for url 'https://neuragrowth.co/blog/schema-grammar-ceiling/' 521 is Cloudflare saying the origin server did not answer. So the interesting failure was not in the syndication job at all. My whole site was down, and had been for over three hours by then. The server itself was fine: four days of uptime, load under 0.2, disk at eight percent. But systemctl is-active nginx said failed , and nothing was listening on 80 or 443. nginx resolves your upstreams before it starts The journal had the whole thing in three lines: 06:49:54 systemd[1]: Stopping nginx.service... 06:49:54 nginx[36027]: [emerg] host not found in upstream "example-backend.tld" in /etc/nginx/sites-enabled/site:104 06:49:54 nginx[36027]: nginx: configuration file test failed Line 104 was a small proxy I had added months earlier so the public site could forward one form endpoint to a backend on a different host without revealing its name: location = /api/lead-capture { proxy_pass https://example-backend.tld/api/lead-capture ; proxy_ssl_server_name on ; proxy_set_header Host example-backend.tld ; } When proxy_pass contains a literal hostname, nginx resolves it while parsing the configuration , and treats failure as a fatal config error. That resolution happens inside ExecStartPre=/usr/sbin/nginx -t , so a name it cannot look up means the unit never starts. The config was not wrong. It was valid before the restart and valid after, and nginx -t passed by hand seven hours later. It was invalid for about one second. Why DNS was gone for exactly that instant Ten seconds of journal, reconstructed: 06:49:44 apt-daily-upgrade.service starts 06:49:53 "Reexecution requested ... (unit apt-daily-upgrade.service)" 06:49:53 systemd reexecuting (it had just upgraded itself) 06:49

Disclosure: I work on Tessera, which is one of the tools in the gap I am describing. Ask a fifty-person engineering organisation how they control production access and you will hear the same answer with small variations: a bastion host, SSH keys distributed by configuration management, a shared kubeconfig somewhere, and a spreadsheet or a Notion page that is out of date. Nobody chose that. It is what remains after the alternatives were priced. How the category got shaped Privileged access management grew up serving banks, telcos and governments in the 2000s. Those buyers had specific characteristics: thousands of administrators, regulators with written opinions, dedicated security teams, and procurement processes measured in quarters. Products shaped themselves accordingly. Six-figure entry prices. Deployments measured in months with professional services attached. Feature sets covering every mainframe and network appliance in a bank's estate. Sales motions that start with a discovery call and a mutual NDA. That was a reasonable fit for those buyers. It is a terrible fit for a company with sixty engineers, no dedicated security team, one person who is security-adjacent, and a procurement process that consists of a founder approving a card payment. So the mid-market did what people do when a category prices them out: they built the minimum themselves. A bastion is a bastion because it was free and it was Tuesday. The problem this leaves The bastion answer works, up to a point, and it is worth being specific about where the point is. A bastion controls the door. It does not control the room. Once someone is through, there is no per-command record, no way to reduce their privileges while they are working, and no session replay. And keys still have to be distributed and revoked behind it, which means the original problem is intact — it just has a nicer front entrance. Three things then converge, usually in the same year: The first enterprise customer. Their security que

Disclosure: I work on an access tool (Tessera), mentioned once at the end. Everything before that is about evidence, and applies whatever you use. The most common surprise in a first SOC 2 or ISO 27001 audit is not that a control is missing. It is that a control exists, works, and cannot be evidenced — so it counts as absent. The distinction is worth stating precisely, because it is not obvious until it has cost you something. A control is a thing that is true about your system. Only authorised engineers can reach production. Evidence is an artefact, produced by a system rather than by a person, that demonstrates the control was operating throughout the audit period — not on the day someone checked. Most organisations have decent controls. Most cannot produce evidence, because their controls live in places that do not emit artefacts: a bastion's authorized_keys file, a spreadsheet, a Slack thread where someone approved something, and the collective memory of three engineers. What gets asked for Reconstructed from what people have told me, this is the shape of the questions: "Show me everyone who could access production on 14 March." Not today. A specific date in the past, usually chosen by the auditor. This is the one that catches people, because most systems can tell you the current state and cannot tell you a historical one. authorized_keys has no history. A spreadsheet has whatever history git gives it, if it is in git, which it usually is not. "Show me that this person's access ended when their employment ended." Both timestamps, from two systems, matched. HR has the first. The second is the problem. "Show me the approval for this elevated access." Not that a policy requires approval — the specific approval, for this specific grant, with who approved it and when. "Show me what was done in this session." Increasingly common where production access to customer data is involved. Not "we log commands", but the actual record for a named session. "Show me that these c

Disclosure: I work on Tessera, which is self-hosted. That is the position I am arguing from, and the costs of that position are in the last section. Security questionnaires ask where customer data is processed. Access-control tools tend to get a shallow answer to that question, because people think of them as gatekeepers rather than as data processors. They are both. Here is what a hosted access broker necessarily knows about you. Your infrastructure inventory. Every registered target: hostnames, addresses, cluster endpoints, database names, environment labels. That is a map of your estate, and it is the document an attacker would most like to read before deciding where to spend effort. Your organisational structure. Who has access to what, which teams exist, who approves for whom, who was granted production access at 2am during an incident. Org charts are inferable from access graphs with unpleasant accuracy. Your session content, if recording is on. Every command, every query, every screen of output. That includes whatever your engineers pasted into a terminal, which — be honest about your own estate here — includes secrets sometimes. Timing. When your incidents happen, how long they last, who gets pulled in. That is commercially sensitive on its own. None of that requires the vendor to hold your credentials. It is the metadata, and the metadata is the part that survives every architectural mitigation. Where this shows up GDPR and processor chains. Session recordings contain personal data — identified individuals performing identified actions at identified times. A hosted vendor is a processor, which means a DPA, transfer mechanisms if data leaves the EEA, and a sub-processor list you have to monitor. It also means their sub-processors become your problem, which is a chain you do not control and cannot easily audit. Sector rules. Financial services, healthcare and public sector procurement in most European jurisdictions have specific requirements about where data

Disclosure: I work on Tessera, which is on the buy side of this. I have tried to cost the build side properly, because a comparison where the build option looks stupid is a comparison nobody believes. Every engineering team that has this problem considers building it. That instinct is correct. SSH certificates are a well-understood, well-documented mechanism, everything you need ships with OpenSSH, and the first working version takes a competent engineer about a week. The week is not the cost. Here is what is. The build, honestly The CA itself. You generate a key pair, configure targets with TrustedUserCAKeys , and sign user keys with a short validity. This part genuinely is a week, and it works. Protecting the CA key. This is where the estimate starts moving. The CA private key can now grant access to every host in the fleet. On a laptop it is an incident waiting to happen. So you want it in an HSM or a KMS, which means an integration, which means the signing operation now has an availability dependency, which means a runbook for what happens when that dependency is down. Call it two to four weeks including the operational work. Issuance. Someone has to request a certificate and something has to decide whether to give them one. That means integrating with your identity provider, mapping groups to principals, building a request path, and building an approval path if you want anything time-boxed or justified. This is the real project, and it is measured in months rather than weeks, because it is where the requirements keep arriving. Rollout across the estate. Editing sshd_config on every production host. Technically trivial, organisationally not: change window, sign-off, rollback plan, and the discovery that four hosts are not in configuration management and one of them is important. Audit. Certificates tell you a session was authorised. They do not tell you what happened in it. If your requirement includes per-command history or session replay — and if you have audi

Disclosure: these are decisions from Tessera, which I work on. They are all small enough to copy into your own service, which is why they are worth writing up. Security feature lists are made of nouns: encryption, RBAC, SSO, audit. The things that actually decide whether a system holds up are smaller than that and never make the list. Here are four of ours, with the reasoning, including the case where we made the trade in the direction most people do not. 1. The login rate limit ignores X-Forwarded-For Rate limiting a login endpoint is table stakes. The question is what you count against. The natural implementation reads X-Forwarded-For , because your service is behind a load balancer and the real client address is in that header. Almost every tutorial does it this way. The problem: X-Forwarded-For is a request header. If your service trusts it, an attacker sets a different value on every request and each one gets its own bucket. You have not built a rate limit, you have built a counter that resets on demand, and the dashboards will look perfectly healthy while the endpoint is being brute-forced. We count against the real TCP connection address instead. That is the peer address of the socket, which an attacker cannot forge without actually controlling that address. The cost is real and worth naming. Behind a reverse proxy, every request arrives from the proxy's address, so the limit applies to the proxy as a whole rather than per client. That is a worse experience in some topologies, and people will file bugs about it. We took that trade because a rate limit that can be bypassed by setting a header is not a degraded rate limit, it is the absence of one — and the absence is worse than useless, because it looks like presence. If you do want per-client limits behind a proxy, the answer is an explicit allowlist of trusted proxy addresses whose forwarded headers you accept, not blanket trust of a header. 2. Target addresses are validated to prevent SSRF Our controller is