标签：#infra

A practical framework for test planning, hardware scheduling, artifact traceability, failure classification, and evidence-based quality gates Disclaimer: The views expressed in this article are my own. The architecture, examples, terminology, and code snippets are generalized for educational purposes and do not describe or disclose any employer’s proprietary systems, confidential information, or internal implementation details. A software change can compile successfully, pass unit tests, and still introduce a serious GPU regression. The failure may appear only on one GPU generation. It may depend on a particular driver, firmware revision, operating system, graphics API, or workload. A change may preserve functional correctness while quietly reducing performance. It may also cause an intermittent failure that disappears when the test is rerun. This is why GPU validation cannot be treated as conventional CI/CD with a GPU runner attached to the end of the pipeline. A dependable GPU validation platform must coordinate: Software and firmware artifacts Hardware configurations Test coverage GPU resource scheduling Failure classification Performance baselines Engineering evidence It must do all of this while operating under an important constraint: compatible GPU capacity is limited and expensive. The objective is not simply to run more tests. It is to produce reliable evidence quickly enough to support engineering decisions. Why conventional CI/CD is not enough A conventional application pipeline often resembles: Commit ↓ Build ↓ Unit tests ↓ Integration tests ↓ Deployment A GPU validation pipeline is more multidimensional: Code or configuration change ↓ Build software and firmware artifacts ↓ Determine affected GPU configurations ↓ Reserve compatible hardware ↓ Prepare the driver and runtime environment ↓ Run functional, stability, and performance tests ↓ Collect logs, traces, metrics, and crash artifacts ↓ Classify failures and compare results with baselines ↓ Make a mer

Max Korbacher explains why successful internal development platforms cannot be built on tech alone. He discusses the pitfalls of infrastructure-first thinking, the importance of a clear product mindset, and how to measure real value using DevEx and SPACE metrics. Learn how to align your team, manage tech debt, and foster a thriving community to ensure lasting platform adoption. By Max Körbächer

Originally published at blog.whynext.app . It started as an ordinary cleanup problem. Users upload media files (recordings and images) through presigned URLs. The server issues an upload URL, the client uploads straight to S3, then calls a commit API to say "register this key as a real asset." The problem is what happens when someone gets a presign but never commits. The app crashes, the network drops, the user leaves the screen, and the bucket is left with an object that isn't registered anywhere. I wanted a lifecycle rule to clean these up, but there was no way to write one. Committed and uncommitted objects were mixed under the same prefix, so any rule that says "delete old things" would delete real assets too. A daily upload quota kept the pile from growing fast, but the fact remained: there was no path to reclaim the space. The design: what isn't committed lives in tmp The backbone of the fix is key namespace separation. presign issues a temporary key under the tmp/ prefix. When commit passes validation (existence check via HEAD, Content-Type, size limit), it promotes the object to its final key with CopyObject and deletes the tmp original. Objects whose commit never arrives stay in tmp/ , and a lifecycle rule expires them after 7 days. Now the lifecycle rule only has to look at tmp/ . Real assets are outside its blast radius from the start. The clients didn't need to change. I read all three upload flows to confirm this: every one of them uses the key returned in the commit response for its follow-up calls, so the server can change the key shape without them noticing. Commits for old-format keys already in flight at deploy time still go through the existing path. One trap here. This bucket has versioning enabled. On a versioned bucket, expiration doesn't delete an object. It only adds a delete marker, and the original bytes stay behind as a noncurrent version. Without a paired noncurrent_version_expiration (1 day), the cleanup runs and not a single byte is rec

AWS has recently introduced CloudFormation express mode, a deployment option that can reduce infrastructure deployment times by marking stack operations complete once resource configuration is applied, rather than waiting for full resource stabilization. By Renato Losio

There is a note in my runbooks that says the Enter key works. That is almost the whole note. Press Enter and the prompt submits. I wrote it in bold, with sources and a date, because one of my AI agents once fixed the Enter key. The fix was the bug. The Fix Was the Bug I run a fleet of Claude Code agents in tmux panes. An orchestrator script types a prompt into a pane and presses Enter for it. One day an agent decided the submissions were not going through, and it knew the fix: send a backslash before the Enter. That sounds plausible if you have ever fought a terminal. It is also exactly backwards. In Claude Code, backslash plus Enter inserts a newline. It is the multiline key. So the fix turned every prompt into a draft that quietly grew longer and never sent. The panes looked busy. Nothing was happening. The submissions had been fine all along. The agent just had not waited for the reply. What broke: An agent changed Enter to backslash-Enter to repair prompt submission. Backslash-Enter creates a newline, so the repair silently stopped every prompt from sending. The system it fixed had never been broken. There is a second note in the same genre. My monitoring dashboard answers its health check with a 302 redirect to the login page. That is what healthy looks like there. But a 302 looks like trouble if you are hunting for outages, and agents kept trying to repair it. So the runbook now says, more or less: the redirect is normal, do not fix the healthy system. Human teams do not write notes like these. Nobody pins "the Enter key works" to the wall of an office. A New Hire Every Session Why did I have to? Because a human hits a dead end once. The wince is the documentation. Whoever broke a build by fixing the Enter key would remember it for years, tell the story at lunch, and the whole team would absorb it without anyone writing a word. An agent has none of that. It has no episodic memory. Every fresh context window is a new hire: smart, fast, and seeing your system fo

QCon AI Boston 2026 focused on the operational challenges of deploying AI agents, emphasizing the need for robust production infrastructure. Key themes included improving context management, ensuring security through a "harness" around agents, and adopting a comprehensive engineering model for AI. By Tatiana Fesenko

El ecosistema de agentes autónomos está evolucionando. Mientras la atención se centra en modelos y frameworks, empiezan a aparecer proyectos que cubren necesidades operativas básicas: comunicación, almacenamiento, finanzas. Aquí van cinco que vale la pena observar. 1. Apumail: el correo nativo para agentes Apumail ofrece direcciones de email que los agentes pueden crear y leer mediante una API REST plana. El contenido se negocia automáticamente: texto plano para agentes, HTML para humanos. Señal relevante: es el primer intento serio de darle a un agente un buzón de correo con el mismo estándar que usamos los humanos, sin adaptadores. Si los agentes empiezan a gestionar correspondencia, este tipo de servicio será indispensable. 2. RogerThat: chat entre agentes Una capa de mensajería en tiempo real diseñada para que agentes conversen entre sí. RogerThat no es un chat humano con bots, sino un canal donde los agentes coordinan acciones. Contexto: si varios agentes intervienen en un mismo workflow, necesitan un bus de eventos. RogerThat plantea que ese bus puede ser un chat, con las garantías de entrega y orden que eso implica. 3. DOBI: agente para DePIN y activos del mundo real Agent autónomo que opera sobre la cadena para gestionar infraestructuras físicas descentralizadas (DePIN). Ejecuta acciones on-chain a partir de decisiones tomadas por el modelo. Patrón: no es un simple bot de trading; apunta a mantenimiento de equipos, comprobación de sensores, distribución de incentivos. La frontera entre software y hardware se desdibuja. 4. CIDIF: financiamiento de I+D para agentes Plataforma que automatiza la presentación y seguimiento de solicitudes a fondos de innovación. El agente rellena formularios, adjunta documentación y trackea el estado. No obvio: la burocracia gubernamental es un entorno altamente estructurado (pocas decisiones abiertas, muchos campos fijos). Es un terreno ideal para agentes, aunque el ruido político lo opaque. 5. Orquesta: orquestación de flujos mu

GDPR Compliance Fails When It Exists Only in Policy Documents Many organizations can produce a privacy policy, a data processing register, and a set of security procedures. The harder question is whether their infrastructure can actually protect, recover, trace, and report personal data when something goes wrong. GDPR compliance is often discussed as a legal project. In practice, many of its most difficult requirements depend on everyday IT operations. Can the organization recover personal data after a destructive incident? Can it identify who restored, copied, accessed, or deleted a backup? Can it detect suspicious activity quickly enough to support an investigation? Can it demonstrate that protection controls are applied across on premises, cloud, and hybrid environments? Policies explain intent. Operational controls provide evidence. Availability Is a Privacy Requirement Too Privacy discussions often focus on unauthorized access and disclosure. Data loss and prolonged unavailability matter as well. Personal data may become unavailable because of ransomware, storage failure, accidental deletion, database corruption, software defects, or a regional outage. If the organization cannot restore that data, it may be unable to serve customers, respond to data subject requests, or maintain essential business processes. A GDPR aligned protection strategy should therefore include reliable backup, offsite copies, tested recovery, and resilience across multiple failure scenarios. The backup architecture should support the systems that actually contain personal data, including databases, virtual machines, file servers, object storage, cloud workloads, and large data platforms. Protecting only the most visible applications leaves hidden gaps. Encryption Is Necessary, but It Is Not the Whole Answer Encryption protects data during transmission and storage, especially when backup copies move across networks or are stored outside the production environment. However, encrypted data

You've probably registered a domain with Route53 or another popular registrar. You pick something...

The dashboard says forty instances, up from twelve this morning. The autoscaler did its job: it saw latency climb and threw hardware at it. And latency got worse. Not flat. Worse. You're paying for three times the compute to serve a slower product. Somewhere under all forty of those boxes is a single thing they're all waiting in line for, and every instance you add makes the line longer. Horizontal scaling multiplies work that doesn't have to coordinate. The instant the work does have to coordinate, more instances make it slower. Amdahl wrote this down in 1967: the serial fraction of a job sets a hard ceiling on how much faster you can go, no matter how much hardware you throw at the parallel part. Neil Gunther's Universal Scalability Law goes further: past a certain point, the cost of nodes coordinating with each other bends the curve back down. Add capacity, get less throughput. That ceiling was not set by the autoscaler, and it will not be moved by the autoscaler. It was set a long time before this morning, in a room, by whoever decided where the state lives and who has to touch it at the same instant. Now hand the service to a fleet of agents. It writes you something that looks built to scale: stateless handlers, a tidy repo, green tests, a canary that bakes fine at 1% traffic. Every gate you trust says ship it. And the bottleneck is sitting right there in the design, invisible to all of it, because the mistake isn't in the lines, it's in the shape. You cannot catch a shape problem by reading a diff. Name the hot state before you pick a framework. Where does the contended state live, and which requests touch it at the same instant? Answer that out loud, before anyone opens an editor. The tool is downstream of that answer, every time. Originally published at https://imacto.com/writing/scale-is-a-design-not-a-dial . Written with Claude Opus 4.8.

“We have no choice but to address the challenges created by these massive facilities,” New York governor Kathy Hochul said. The executive order will pause construction for one year.

I didn’t set out to diagnose hyperscalers. I wasn’t doing a grand industry analysis. I wasn’t mapping global architecture. I wasn’t trying to understand cloud strategy. I was just trying to use a popular software provider — and everything kept breaking. Every time something failed, I followed the thread. And every thread led to the same architectural gap. Eventually I realised I hadn’t been analysing hyperscalers at all. I’d accidentally mapped the substrate failure across the entire industry. Once you see the pattern, you can’t unsee it. Across Microsoft, AWS, Google, and Meta, the same structural drift appears: meaning drift identity drift trust drift state drift execution drift provenance drift agentic drift Different companies. Different stacks. Different histories. Same substrate gap. And it’s not just me. The world is waking up to these problems too. Vendor lock in isn’t just a technical nuisance anymore — it’s becoming a public conversation. People are asking why their money keeps disappearing into the same handful of providers. Organisations are asking why their systems collapse the moment they try to leave. Governments are asking why critical infrastructure depends on architectures they cannot inspect, cannot govern, and cannot reproduce. What started as a personal frustration with a popular software provider turns out to be the same structural issue everyone else is now discovering. And sovereignty is entering the conversation — not as a political slogan, but as an architectural question. When national systems depend on fragmented substrates owned by a tiny cluster of vendors, sovereignty becomes a structural issue. The question isn’t “who controls the cloud?” It’s “who controls the substrate the cloud is built on?” Follow the thread far enough and you reach a scenario nobody wants to think about: what happens in a moment of global stress when a hyperscaler’s fragmented substrate becomes a single point of failure? Not a political crisis — a structural one.

The Cybersecurity and Infrastructure Security Agency (CISA) has issued a postmortem on a data leak in which a contractor published dozens of internal CISA credentials -- including AWS Govcloud keys -- in a public GitHub repository for almost six months before being notified by KrebsOnSecurity. Experts say the gaps identified in the agency's initial response provide important lessons that all security teams should absorb.

Bryan Oliver discusses the frontier of AI infrastructure: chaos engineering for large-scale GPU clusters. He shares how engineering leaders can handle complex topologies, network protocols like RDMA, and NUMA misalignments. Discover seven practical fault-injection strategies to maximize multi-million dollar hardware efficiency and build robust observability loops. By Bryan Oliver

The company is taking a modular approach to designing these chips, anticipating that their needs will change as AI evolves rapidly by the time the chips are in production.

In version 1.11, HashiCorp introduced Terraform Ephemeral resources and write-only attributes to allow for root configs that do not store secrets in the Terraform statefile. But many users ask about how they can adopt ephemerals. This blog attempts to lay out the ways secrets can be stored in state and how you should update your configurations to remove those secrets. Note: For a primer on ephemerals ( see this blog post ). Scenarios to consider: Data sources that fetch a static secret Resources that receive a secret Resources that generate a dynamic a secret Resources that fetch generated secrets to store in another 3rd party system Scenario 1: Data sources with static secrets Ephemeral resources can often be a drop-in replacement for data sources pulling static values: data "vault_kv_secret_v2" "static_kv" { mount = "kvv2" name = "my_password" } ephemeral "vault_kv_secret_v2" "static_kv" { mount = "kvv2" name = "my_password" } However, using these values has 1 specific difference. The attributes on a ephemeral resource are considered ephemeral and can only be used as ephemeral arguments. That means 2 places: Provider blocks Provider blocks are considered ephemeral, so ephemeral resources may populate arguments: provider "example" { password = tostring ( ephemeral . vault_kv_secret_v2 . static_kv . data . password ) } Write-only arguments Write-only arguments are special arguments that require the ephemeral taint for values: resource "aws_db_instance" "example" { ... password_wo = tostring ( ephemeral . vault_kv_secret_v2 . static_kv . data . password ) } If the resource you wish to pass a value to does not have an available ephemeral, open an issue with that provider. You can reference: this blog post this agent skill Scenario 2: Resources that receive a static secret Without duplicating to the section above, write-only arguments are a way to get secrets out of state. Above has guidance if the secret value comes from a data source, but what if its from a variable?

AWS has outlined how ProGlove, an industrial-wearables manufacturer, was able to scale its SaaS platform to run more than one million AWS Lambda functions spread across thousands of dedicated customer accounts. By Matt Foster

Static analysis isn't just for application source code. Terraform, Pulumi, OpenTofu, and CloudFormation files are code too — and they get misconfigured just as often as a backend service. A public S3 bucket, a security group open to 0.0.0.0/0 , or an unencrypted RDS instance are all bugs you can catch before apply ever runs. TFSec is the tool most people reach for first, but it's not the only option on the OWASP Source Code Analysis Tools list . In this article I'll use Checkov , a free, open-source policy-as-code scanner built by Bridgecrew (now part of Palo Alto Networks), to scan a Terraform project end to end — from a local scan to a GitHub Actions gate that blocks merges on critical misconfigurations. The same approach works with OpenTofu and Pulumi projects too, since Checkov understands HCL directly and also has native support for Pulumi's rendered plan output and CloudFormation/ARM/Kubernetes manifests. Why Checkov? 100% open source (Apache 2.0), actively maintained, thousands of built-in policies. Understands Terraform, OpenTofu, CloudFormation, Kubernetes, Helm, Dockerfile, ARM, Serverless Framework, and Pulumi (via cdktf /synthesized plans) — one tool across most of your IaC surface. No account or API key required to run locally or in CI. Supports custom policies written in Python or YAML if the built-in rule set doesn't cover something specific to your org. ## 1. The sample infrastructure A small AWS setup with a few intentionally introduced misconfigurations — the kind that get merged during a rushed sprint: # main.tf provider "aws" { region = "us-east-1" } resource "aws_s3_bucket" "data" { bucket = "company-app-data-bucket" } # Vulnerable: bucket has no encryption, no versioning, and is publicly readable resource "aws_s3_bucket_acl" "data_acl" { bucket = aws_s3_bucket . data . id acl = "public-read" } resource "aws_security_group" "web" { name = "web-sg" description = "Allow web traffic" # Vulnerable: SSH open to the entire internet ingress { from_port

Vermilion Cliffs Ventures announced Wednesday the close of a $25 million Fund II.

Why Every Modern SaaS Needs a Billing Engine (Not Just a Payment Gateway) When building a SaaS product, API platform, AI application, or cloud service, most teams start by integrating a payment gateway like Stripe or Paystack. That solves one important problem: Collecting payments. But collecting payments and managing billing are two completely different challenges. As products grow, pricing becomes more sophisticated. Customers expect usage-based billing, flexible subscriptions, prepaid credits, seat-based pricing, tiered plans, and transparent invoices. Suddenly, the billing logic inside the application becomes one of the most complex parts of the entire system. This is why more engineering teams are separating payment processing from billing infrastructure . Payment Processing vs Billing Infrastructure A payment gateway is responsible for moving money. A billing platform is responsible for determining what should be charged, when it should be charged, and why it should be charged. Although these responsibilities often get grouped together, they solve very different problems. Payment providers typically handle: Payment authorization Card processing Refunds Payment retries Bank settlements Customer payment methods A billing engine manages: Usage metering Subscription billing Pricing rules Billing cycles Invoice generation Revenue analytics Customer usage tracking Payment orchestration Multi-gateway billing Proration Credits and quotas Keeping these responsibilities separate makes systems easier to maintain, easier to scale, and much more flexible when pricing changes. Why Billing Logic Doesn't Belong Inside Your Application Many engineering teams start with a simple subscription model. Then the business evolves. Suddenly the product needs to support: API usage billing AI token billing Storage billing Compute billing Per-seat pricing Hybrid subscriptions Enterprise contracts Prepaid credits The application slowly becomes filled with billing-specific code. Developers