标签：#infrastructure

Jörg Schad explains how to tame the complex "data management hairball" to build scalable, safe architectures for AI. He shares how autonomous data products act like containers for data, encapsulating pipelines, schemas, and metadata. Discover how progressive tool discovery via protocols like MCP limits context rot, enforces governance policies, and ensures reliable, multi-modal access. By Jörg Schad

Moonshot launched Kimi K3, and within 48 hours it had to stop taking new subscribers. Not because the model flopped, but because too many people wanted it. That's a strange kind of problem to have, and it's telling you something important about where AI's real bottleneck now sits. What actually happened Less than two days after Kimi K3 went live, Moonshot froze new subscriptions. The reason was blunt: demand had eaten through its available GPU capacity. In the company's own words, the model got "far more love than we expected," and in 48 hours usage pushed close to the limit of what its hardware could serve. Moonshot handled it reasonably. Existing users kept their access, and the company said it would expand capacity and reopen signups in batches. It also split its plans into two tiers, a general Kimi Membership for web and app use, and a separate Kimi Code Membership aimed at programming work. That split is a hint about which users are burning the most compute. Why the model drew that kind of demand Kimi K3 isn't a minor release. It's an open-weight model at 2.8 trillion parameters, and it reportedly beat Anthropic's Fable 5 and OpenAI's GPT-5.6 Sol on front-end coding tests. Open, cheap, and competitive on real coding is exactly the combination developers pile onto. So they did. The real story: the bottleneck moved Here's the part worth internalizing. For years the hard, expensive problem in AI was training. That's where the giant compute bills and the headlines were. Kimi K3's freeze shows the constraint shifting to inference, the cost of actually running the model for users, every request, every day. Agentic workloads are why. When an AI agent runs a coding task for minutes or hours instead of answering a single prompt, each user consumes far more compute than a chatbot ever did. Multiply that by a viral launch and you hit a GPU wall fast. Moonshot didn't run out of ideas. It ran out of chips to serve the ideas. Why this matters even if you never touch Kimi Thi

Hetzner is experimenting with LLM inference. That is not a sentence I expected to write, but I think it is pretty interesting :) Before anyone moves their production AI workloads to Hetzner: this is very much an experiment . There is no billing, no SLA, no production guarantee, and currently only one model. Hetzner says it wants to learn whether people actually want this, how the system scales, which features matter, and what kind of load it can handle. So this is not a finished product launch. It is Hetzner putting something early in front of users and seeing what happens. I really like that approach. What Is Hetzner Inference? Hetzner Inference is an OpenAI-compatible API running on Hetzner's own infrastructure. You create an API token in the Experiments dashboard, point an OpenAI client at Hetzner's base URL, and use it like most other inference APIs. Right now, the only available model is Qwen/Qwen3.6-35B-A3B-FP8 . It is a 35-billion-parameter Mixture-of-Experts model with 3 billion active parameters. It accepts text and images, has a 262K context window, and uses FP8-quantized weights. That is a perfectly reasonable model for an experiment. It is small enough to serve without a ridiculous GPU cluster, but still useful enough to test the API with real workloads. Hetzner also published a short tutorial for connecting OpenCode to the API , if you want to try it without writing any code. I Tried It Because the API is OpenAI-compatible, there is almost nothing special about the integration: pip install openai from openai import OpenAI client = OpenAI ( base_url = " https://inference.hetzner.com/api/v1 " , api_key = " YOUR_TOKEN " , ) response = client . chat . completions . create ( model = " Qwen/Qwen3.6-35B-A3B-FP8 " , messages = [ { " role " : " user " , " content " : " Explain why the sky is blue in one sentence. " } ], extra_body = { " chat_template_kwargs " : { " enable_thinking " : False , } }, ) print ( response . choices [ 0 ]. message . content ) The enabl

An updated government advisory warns that Iranian hackers are exploiting systems used by water and energy providers.

A practical framework for test planning, hardware scheduling, artifact traceability, failure classification, and evidence-based quality gates Disclaimer: The views expressed in this article are my own. The architecture, examples, terminology, and code snippets are generalized for educational purposes and do not describe or disclose any employer’s proprietary systems, confidential information, or internal implementation details. A software change can compile successfully, pass unit tests, and still introduce a serious GPU regression. The failure may appear only on one GPU generation. It may depend on a particular driver, firmware revision, operating system, graphics API, or workload. A change may preserve functional correctness while quietly reducing performance. It may also cause an intermittent failure that disappears when the test is rerun. This is why GPU validation cannot be treated as conventional CI/CD with a GPU runner attached to the end of the pipeline. A dependable GPU validation platform must coordinate: Software and firmware artifacts Hardware configurations Test coverage GPU resource scheduling Failure classification Performance baselines Engineering evidence It must do all of this while operating under an important constraint: compatible GPU capacity is limited and expensive. The objective is not simply to run more tests. It is to produce reliable evidence quickly enough to support engineering decisions. Why conventional CI/CD is not enough A conventional application pipeline often resembles: Commit ↓ Build ↓ Unit tests ↓ Integration tests ↓ Deployment A GPU validation pipeline is more multidimensional: Code or configuration change ↓ Build software and firmware artifacts ↓ Determine affected GPU configurations ↓ Reserve compatible hardware ↓ Prepare the driver and runtime environment ↓ Run functional, stability, and performance tests ↓ Collect logs, traces, metrics, and crash artifacts ↓ Classify failures and compare results with baselines ↓ Make a mer

Max Korbacher explains why successful internal development platforms cannot be built on tech alone. He discusses the pitfalls of infrastructure-first thinking, the importance of a clear product mindset, and how to measure real value using DevEx and SPACE metrics. Learn how to align your team, manage tech debt, and foster a thriving community to ensure lasting platform adoption. By Max Körbächer

Originally published at blog.whynext.app . It started as an ordinary cleanup problem. Users upload media files (recordings and images) through presigned URLs. The server issues an upload URL, the client uploads straight to S3, then calls a commit API to say "register this key as a real asset." The problem is what happens when someone gets a presign but never commits. The app crashes, the network drops, the user leaves the screen, and the bucket is left with an object that isn't registered anywhere. I wanted a lifecycle rule to clean these up, but there was no way to write one. Committed and uncommitted objects were mixed under the same prefix, so any rule that says "delete old things" would delete real assets too. A daily upload quota kept the pile from growing fast, but the fact remained: there was no path to reclaim the space. The design: what isn't committed lives in tmp The backbone of the fix is key namespace separation. presign issues a temporary key under the tmp/ prefix. When commit passes validation (existence check via HEAD, Content-Type, size limit), it promotes the object to its final key with CopyObject and deletes the tmp original. Objects whose commit never arrives stay in tmp/ , and a lifecycle rule expires them after 7 days. Now the lifecycle rule only has to look at tmp/ . Real assets are outside its blast radius from the start. The clients didn't need to change. I read all three upload flows to confirm this: every one of them uses the key returned in the commit response for its follow-up calls, so the server can change the key shape without them noticing. Commits for old-format keys already in flight at deploy time still go through the existing path. One trap here. This bucket has versioning enabled. On a versioned bucket, expiration doesn't delete an object. It only adds a delete marker, and the original bytes stay behind as a noncurrent version. Without a paired noncurrent_version_expiration (1 day), the cleanup runs and not a single byte is rec

AWS has recently introduced CloudFormation express mode, a deployment option that can reduce infrastructure deployment times by marking stack operations complete once resource configuration is applied, rather than waiting for full resource stabilization. By Renato Losio

There is a note in my runbooks that says the Enter key works. That is almost the whole note. Press Enter and the prompt submits. I wrote it in bold, with sources and a date, because one of my AI agents once fixed the Enter key. The fix was the bug. The Fix Was the Bug I run a fleet of Claude Code agents in tmux panes. An orchestrator script types a prompt into a pane and presses Enter for it. One day an agent decided the submissions were not going through, and it knew the fix: send a backslash before the Enter. That sounds plausible if you have ever fought a terminal. It is also exactly backwards. In Claude Code, backslash plus Enter inserts a newline. It is the multiline key. So the fix turned every prompt into a draft that quietly grew longer and never sent. The panes looked busy. Nothing was happening. The submissions had been fine all along. The agent just had not waited for the reply. What broke: An agent changed Enter to backslash-Enter to repair prompt submission. Backslash-Enter creates a newline, so the repair silently stopped every prompt from sending. The system it fixed had never been broken. There is a second note in the same genre. My monitoring dashboard answers its health check with a 302 redirect to the login page. That is what healthy looks like there. But a 302 looks like trouble if you are hunting for outages, and agents kept trying to repair it. So the runbook now says, more or less: the redirect is normal, do not fix the healthy system. Human teams do not write notes like these. Nobody pins "the Enter key works" to the wall of an office. A New Hire Every Session Why did I have to? Because a human hits a dead end once. The wince is the documentation. Whoever broke a build by fixing the Enter key would remember it for years, tell the story at lunch, and the whole team would absorb it without anyone writing a word. An agent has none of that. It has no episodic memory. Every fresh context window is a new hire: smart, fast, and seeing your system fo

QCon AI Boston 2026 focused on the operational challenges of deploying AI agents, emphasizing the need for robust production infrastructure. Key themes included improving context management, ensuring security through a "harness" around agents, and adopting a comprehensive engineering model for AI. By Tatiana Fesenko

GDPR Compliance Fails When It Exists Only in Policy Documents Many organizations can produce a privacy policy, a data processing register, and a set of security procedures. The harder question is whether their infrastructure can actually protect, recover, trace, and report personal data when something goes wrong. GDPR compliance is often discussed as a legal project. In practice, many of its most difficult requirements depend on everyday IT operations. Can the organization recover personal data after a destructive incident? Can it identify who restored, copied, accessed, or deleted a backup? Can it detect suspicious activity quickly enough to support an investigation? Can it demonstrate that protection controls are applied across on premises, cloud, and hybrid environments? Policies explain intent. Operational controls provide evidence. Availability Is a Privacy Requirement Too Privacy discussions often focus on unauthorized access and disclosure. Data loss and prolonged unavailability matter as well. Personal data may become unavailable because of ransomware, storage failure, accidental deletion, database corruption, software defects, or a regional outage. If the organization cannot restore that data, it may be unable to serve customers, respond to data subject requests, or maintain essential business processes. A GDPR aligned protection strategy should therefore include reliable backup, offsite copies, tested recovery, and resilience across multiple failure scenarios. The backup architecture should support the systems that actually contain personal data, including databases, virtual machines, file servers, object storage, cloud workloads, and large data platforms. Protecting only the most visible applications leaves hidden gaps. Encryption Is Necessary, but It Is Not the Whole Answer Encryption protects data during transmission and storage, especially when backup copies move across networks or are stored outside the production environment. However, encrypted data

You've probably registered a domain with Route53 or another popular registrar. You pick something...

The dashboard says forty instances, up from twelve this morning. The autoscaler did its job: it saw latency climb and threw hardware at it. And latency got worse. Not flat. Worse. You're paying for three times the compute to serve a slower product. Somewhere under all forty of those boxes is a single thing they're all waiting in line for, and every instance you add makes the line longer. Horizontal scaling multiplies work that doesn't have to coordinate. The instant the work does have to coordinate, more instances make it slower. Amdahl wrote this down in 1967: the serial fraction of a job sets a hard ceiling on how much faster you can go, no matter how much hardware you throw at the parallel part. Neil Gunther's Universal Scalability Law goes further: past a certain point, the cost of nodes coordinating with each other bends the curve back down. Add capacity, get less throughput. That ceiling was not set by the autoscaler, and it will not be moved by the autoscaler. It was set a long time before this morning, in a room, by whoever decided where the state lives and who has to touch it at the same instant. Now hand the service to a fleet of agents. It writes you something that looks built to scale: stateless handlers, a tidy repo, green tests, a canary that bakes fine at 1% traffic. Every gate you trust says ship it. And the bottleneck is sitting right there in the design, invisible to all of it, because the mistake isn't in the lines, it's in the shape. You cannot catch a shape problem by reading a diff. Name the hot state before you pick a framework. Where does the contended state live, and which requests touch it at the same instant? Answer that out loud, before anyone opens an editor. The tool is downstream of that answer, every time. Originally published at https://imacto.com/writing/scale-is-a-design-not-a-dial . Written with Claude Opus 4.8.

“We have no choice but to address the challenges created by these massive facilities,” New York governor Kathy Hochul said. The executive order will pause construction for one year.

I didn’t set out to diagnose hyperscalers. I wasn’t doing a grand industry analysis. I wasn’t mapping global architecture. I wasn’t trying to understand cloud strategy. I was just trying to use a popular software provider — and everything kept breaking. Every time something failed, I followed the thread. And every thread led to the same architectural gap. Eventually I realised I hadn’t been analysing hyperscalers at all. I’d accidentally mapped the substrate failure across the entire industry. Once you see the pattern, you can’t unsee it. Across Microsoft, AWS, Google, and Meta, the same structural drift appears: meaning drift identity drift trust drift state drift execution drift provenance drift agentic drift Different companies. Different stacks. Different histories. Same substrate gap. And it’s not just me. The world is waking up to these problems too. Vendor lock in isn’t just a technical nuisance anymore — it’s becoming a public conversation. People are asking why their money keeps disappearing into the same handful of providers. Organisations are asking why their systems collapse the moment they try to leave. Governments are asking why critical infrastructure depends on architectures they cannot inspect, cannot govern, and cannot reproduce. What started as a personal frustration with a popular software provider turns out to be the same structural issue everyone else is now discovering. And sovereignty is entering the conversation — not as a political slogan, but as an architectural question. When national systems depend on fragmented substrates owned by a tiny cluster of vendors, sovereignty becomes a structural issue. The question isn’t “who controls the cloud?” It’s “who controls the substrate the cloud is built on?” Follow the thread far enough and you reach a scenario nobody wants to think about: what happens in a moment of global stress when a hyperscaler’s fragmented substrate becomes a single point of failure? Not a political crisis — a structural one.

The Cybersecurity and Infrastructure Security Agency (CISA) has issued a postmortem on a data leak in which a contractor published dozens of internal CISA credentials -- including AWS Govcloud keys -- in a public GitHub repository for almost six months before being notified by KrebsOnSecurity. Experts say the gaps identified in the agency's initial response provide important lessons that all security teams should absorb.

Bryan Oliver discusses the frontier of AI infrastructure: chaos engineering for large-scale GPU clusters. He shares how engineering leaders can handle complex topologies, network protocols like RDMA, and NUMA misalignments. Discover seven practical fault-injection strategies to maximize multi-million dollar hardware efficiency and build robust observability loops. By Bryan Oliver

The company is taking a modular approach to designing these chips, anticipating that their needs will change as AI evolves rapidly by the time the chips are in production.

In version 1.11, HashiCorp introduced Terraform Ephemeral resources and write-only attributes to allow for root configs that do not store secrets in the Terraform statefile. But many users ask about how they can adopt ephemerals. This blog attempts to lay out the ways secrets can be stored in state and how you should update your configurations to remove those secrets. Note: For a primer on ephemerals ( see this blog post ). Scenarios to consider: Data sources that fetch a static secret Resources that receive a secret Resources that generate a dynamic a secret Resources that fetch generated secrets to store in another 3rd party system Scenario 1: Data sources with static secrets Ephemeral resources can often be a drop-in replacement for data sources pulling static values: data "vault_kv_secret_v2" "static_kv" { mount = "kvv2" name = "my_password" } ephemeral "vault_kv_secret_v2" "static_kv" { mount = "kvv2" name = "my_password" } However, using these values has 1 specific difference. The attributes on a ephemeral resource are considered ephemeral and can only be used as ephemeral arguments. That means 2 places: Provider blocks Provider blocks are considered ephemeral, so ephemeral resources may populate arguments: provider "example" { password = tostring ( ephemeral . vault_kv_secret_v2 . static_kv . data . password ) } Write-only arguments Write-only arguments are special arguments that require the ephemeral taint for values: resource "aws_db_instance" "example" { ... password_wo = tostring ( ephemeral . vault_kv_secret_v2 . static_kv . data . password ) } If the resource you wish to pass a value to does not have an available ephemeral, open an issue with that provider. You can reference: this blog post this agent skill Scenario 2: Resources that receive a static secret Without duplicating to the section above, write-only arguments are a way to get secrets out of state. Above has guidance if the secret value comes from a data source, but what if its from a variable?

AWS has outlined how ProGlove, an industrial-wearables manufacturer, was able to scale its SaaS platform to run more than one million AWS Lambda functions spread across thousands of dedicated customer accounts. By Matt Foster