标签：#Architecture

Look at your modern Agentic AI stack. An agent wants to execute a tool, trigger a deployment, access a database, or call an external API. Because nobody fully trusts a probabilistic black box, many teams now use a second probabilistic black box to validate the first one. Think about what is actually happening. You are running hundreds of billions of parameters, consuming tokens, burning GPU resources, and adding hundreds or thousands of milliseconds of latency just to answer a simple operational question: PASS HOLD RED Or in plain English: Continue Verify Stop For many production systems, that's the only decision that matters. Yet we often spend orders of magnitude more compute determining whether an action should execute than executing the action itself. That feels dangerously close to architectural bankruptcy. The Illusion of Prompt-Based Safety We've all done it. You create a prompt: "You are a security validator. If the action appears unsafe, return RED." Then reality arrives. Prompt injections appear. Edge cases appear. Different model versions behave differently. The same input occasionally produces different outputs. And your cloud bill keeps growing. At some point, a difficult architectural question emerges: Can a probabilistic system reliably govern another probabilistic system? Many teams assume the answer is yes. I'm not convinced. The Problem Isn't Intelligence This is where I think the industry may be looking at the problem incorrectly. The challenge is not intelligence. The challenge is governance. LLMs are exceptional at: Reasoning Summarization Code generation Natural language interaction But governance is a different problem. Governance is not asking: "What is the best answer?" Governance is asking: "Should this action be allowed to proceed?" Those are fundamentally different questions. A Different Architecture While exploring this problem, we ended up building a separate deterministic governance layer internally. Instead of generating text, it perf

In November 2023 we ran our first global Hytale servers on Google Kubernetes Engine using Veltrix 3.2 as our configuration orchestrator. The Treasure Hunt Engine—a service that fans spawn to claim event loot—started crashing every time search volume exceeded 12 k RPM. Grafana showed a steady climb of 503 errors on /hunt/claim until the autoscaler maxed out at 32 G1 CPU cores and still couldnt keep up. Operators kept filing tickets that boiled down to one sentence: We click the map, nothing happens. We never saw the actual error because the ingress controller was swallowing it and returning a generic Too many requests. What we tried first (and why it failed) Our first move was to crank up the nginx-ingress-controller replicas from 3 to 12 and switch the load-balancer tier from GKE Standard to Premium. The 503 rate dropped to 8 k RPM, but now the p99 latency on claims spiked from 80 ms to 420 ms. The culprit was a recursive call in the hunt service: every claim required a round trip to the player-profile service to validate tier eligibility, and that service was on a shared Postgres 15.4 cluster with 3 k TPS of unrelated traffic. The error stack in Jaeger was literally tracing_id=7f3a1c8… server=profile-db pool_timeout . We tried adding connection pooling with PgBouncer, but the hunt service was using raw libpq and refused to reuse connections—no matter how many times we told it. The Architecture Decision We ripped the validation out of the synchronous path and made the hunt engine publish an event called HuntTierCheckRequired to a dedicated Kafka topic player-events-tier . The hunt service would respond to the client with a 202 Accepted immediately, then the loot-claim worker would listen to that topic and, if the tier passed, publish HuntLootReady . The worker ran in the same pod but on a separate goroutine with a 60-second TTL so we didnt leak memory if the tier service hung. We moved the player-profile service to an SSD-backed CloudSQL instance and gave it 32 GB R

Google Cloud's automated systems suspended Railway's production account without notice, triggering an eight-hour platform-wide outage affecting 3 million users. The cascade took down workloads across all providers including AWS and bare metal because Railway's control plane was hosted on GCP. Railway is demoting GCP to backup-only status. By Steef-Jan Wiggers

Building a Unix-Domain-Socket IPC server for ECS-on-EC2 services that need to talk fast, cheap, and reliably A while back I was looking at a flamegraph of a service that, on paper, should not have been having any performance problems. The producer and the consumer were the same Docker image's worth of trouble — colocated on the same EC2 host, in the same ECS cluster, sharing the same instance type, the same kernel, the same RAM. By every reasonable measure they were neighbours. And yet every event was making a round trip that looked roughly like this: producer → kernel TCP stack → ENI on the producer task → AWS VPC → internal load balancer → ENI on the consumer task → kernel TCP stack → consumer. TLS handshake. HTTP framing. JSON over the wire. Connection pool. Retry policy. The whole circus. I wasn't doing anything wrong. This is what the platform funnels you toward. ECS with awsvpc networking gives every task its own ENI. The default story for "service A talks to service B" is "give B a DNS name, put a load balancer in front of it, configure a security group, point A at the LB." Even if A and B are physically on the same box, the bytes are still leaving the kernel, traversing the VPC, and coming back. There's a fix for this. It's been a fix for fifty-something years. It just hasn't been the default fix, because cloud-native architecture grew up assuming services would be scattered across hosts and the network was the abstraction that mattered. This article is about building a proper IPC server using Unix Domain Sockets, deployed as a sidecar pattern on ECS-on-EC2, with a wire protocol robust enough to ship in production. We're going to design it from scratch — the transport choice, the wire format, the backpressure model, the failure modes, the deployment topology. I'll show you real pseudo-code from the implementation and call out the small number of places where, if you get it wrong, you'll spend a weekend debugging it. The intended outcome is something you coul

How a shared filesystem became the cheapest, fastest outbox I've ever built — and why FSx for OpenZFS is the version of that idea that finally scales I was staring at an AWS bill last quarter where a single Kinesis Data Streams line item was costing more than the entire S3 footprint sitting behind it. The events on that stream had a useful lifetime of about ninety seconds. They were written by one service, read by another, processed, and dropped. We were paying full streaming-bus price for bytes that barely outlived a TCP timeout. That bill is what got me thinking about transitional data as a category that deserves its own architecture, and about why every "use the right tool" instinct I had — Kinesis, Kafka, MSK — was the wrong tool for this particular shape of work. The right tool, it turns out, is a filesystem. Specifically, AWS FSx for OpenZFS, used as an outbox between producers and consumers, with only a tiny pointer message traveling through whatever messaging bus you already have. This article is the case for that pattern. It's also the design, the failure modes, the code, the cost math, and the honest list of when not to do it. I'll walk you through the architecture from first principles, show you the safe-write protocol that makes it correct under crashes and concurrent retries, compare the cost against Kinesis, MSK and EFS at a realistic petabyte-class workload, and explain why the recent addition of FSx Intelligent-Tiering changes the cost story in a way that makes the pattern attractive even for teams that don't ingest petabytes. If you've ever felt the queasy sensation of paying twice for the same bytes — once to land on a stream, again to land in storage — this is for you. What "transitional data" actually means Most data falls into one of two cleanly shaped buckets. Durable data is the stuff you keep — user records, orders, financial events, audit trails. It needs to live for years; you pay storage costs for those years and you get value over those y

I lost more than 60 days but I made a huge update All OnemanBSD OS is now 64 bits. I also removed all bloat like Rust, Wayland and Qt dependencies from all app ports used in the system. (mpv was changed to gnome-mplayer because of that). New apps included (+ fixed source code of everything): audacity (audio editor) godot (game editor) deadbeef (mp3 player) ted (word processor) Also some bug fixes. Here is the link again: http://bialamusic.com/onemanBSD/ Here are my thoughts: I feel that we as humans are loosing control over technology that was created by humans. This is rising some serious and strange questions. I think we now need tools to power the individual so we can fight back. Not corporations. Not governments. Not organizations. Consider this my bullet in this battle. It is now in your hands so you can be active developers not just passive users.

Martin Fowler's May 27 Fragments brings together four arguments with direct implications for teams working with AI agents. All four are worth covering. Ian Johnson: build quality gates before releasing the agent Ian Johnson published a series about restructuring a gnarly codebase: three months, 258 commits, moving from a Laravel monolith with no tests to an application with automated quality gates and an AI agent shipping production code with minimal supervision. The insight Fowler highlights is about the transition from in-the-loop to on-the-loop: "For the first two months of this project, I used Claude Code with auto-approve turned off. Every file edit, every terminal command, every change… I reviewed it before it executed. The results were good. The code was clean. But I was doing most of the thinking and half the typing. The agent was a fancy autocomplete with better suggestions." Ian Johnson Manual review of every change is not how you build trust in the agent. Trust comes from building the structure that ensures the agent will do the right thing, then stepping back. The sequence: characterization tests first, static analysis, architectural patterns that make things flow correctly. Fowler notes this is exactly the sequence he would use himself. Adam Tornhill: roughly 2 hours of cognitive endurance Adam Tornhill observes that agentic work has a decision density that is mentally more expensive than it appears. The estimate is roughly two hours as a sustainable limit, not a full day of work. The implication: adding more parallel agents does not solve the problem, because the bottleneck is the coordinating engineer's cognitive capacity, not available processing volume. The solutions are smaller tasks, automation, and verification mechanisms, not more parallelism. NHS: closing open source repositories NHS (UK National Health Service) closed open source repositories citing LLM threats to code security. The UK Government Data Services countered directly: making code p

GitHub reports cutting token costs in agentic CI workflows by up to 62% by pruning unused MCP tools, swapping some MCP calls for gh CLI, and running daily “auditor” and “optimizer” agents. A token-usage.jsonl artefact and an Effective Tokens metric help track spend across models and spot regressions. By Mark Silvester

The Problem We Were Actually Solving Our event platform at Veltrix ran a treasure hunt game that gave users real-world rewards. It started as a simple Rails app with a PostgreSQL counter column for each hunt. By 3 AM on Black Friday, that counter column became a single point of failure. Every leaderboard update blocked the entire leaderboard query because PostgreSQL row-level locks escalated to table-level for SERIAL columns. Our error rate jumped from 0.2% to 18% under 2000 concurrent writes. The system didnt just slow down; it started failing writes with could not serialize access due to concurrent update deadlocks. We lost $47K in rewards payouts before we could scale up the database. What We Tried First (And Why It Failed) Our first fix was to shard the PostgreSQL counter by hunt ID, splitting the hot row into 1024 partitions. That reduced the lock contention, but introduced new problems. Each hunt now needed its own sequence, and our Rails code had to route writes to the correct shard. The shard routing introduced 400ms extra latency on leaderboard queries because we had to union results across 1024 tables. Meanwhile, PostgreSQL sequences had gaps up to 1024 when nodes restarted, so our reward payouts were off by thousands on high-traffic hunts. Our Redis cache didnt help because the leaderboard queries were point lookups against 1024 tables, and Redis couldnt pipeline those efficiently. The Architecture Decision We ripped out the PostgreSQL counter and replaced it with a Kafka Streams-based event sourcing system called HuntStream. Every hunt action (point earned, reward claimed) became an immutable event in a Kafka topic. We built a materialized view on top of RocksDB that consumed the topic and maintained the current leaderboard state in memory. The materialized view was partitioned by hunt ID, which meant leaderboard queries only hit one RocksDB partition per hunt. We used RocksDBs built-in caching to keep hot leaderboards in memory, and fall back to disk fo

Everyone is adding AI to their product right now. Most of them are doing it wrong. Not because they chose the wrong model. Not because they used the wrong library. But because they treated AI integration like a regular feature and skipped all the engineering discipline that production systems require. I have integrated LLMs into multiple production applications. This is what I wish I had known before I started. The Mental Model Shift You Need First A traditional API call is deterministic. You send a request, you get a predictable response. You can write tests against it. You can cache it. You can reason about it. An LLM call is not deterministic. The same input can produce different outputs on different runs. The model can refuse, hallucinate, or return output in a format you did not expect. Your system needs to be designed around this reality, not in spite of it. This means defensive parsing, fallback logic, output validation, and graceful degradation are not optional extras. They are the core of the feature. Choosing the Right Model for the Right Job The biggest LLMs are not always the right choice. I learned this building EditDeck Pro, an AI creative platform for music. Some tasks needed a large frontier model for nuanced creative output. Others needed a fast, cheap model that could run many times per session without accumulating significant latency or cost. The pattern that works: Use a lighter model for classification, extraction, and short structured outputs. Use a larger model for generation tasks where quality matters more than speed. Route dynamically between them based on the task type. This can reduce your inference costs by 60 to 80 percent on workloads that mix simple and complex tasks. Prompt Engineering Is Software Engineering Prompts are code. They should be versioned, tested, and reviewed like code. I store prompts in a dedicated module with version numbers. When I change a prompt I run it against a fixed evaluation set of inputs and compare the out

In 1994, Peter Deutsch published the Fallacies of Distributed Computing — eight assumptions that every developer building distributed systems makes, discovers are wrong, and pays for in production. The network is reliable. Latency is zero. Bandwidth is infinite. Each assumption sounds true. Each leads to system failures that could have been avoided. Thirty years later, we're making the same category of mistakes with generative AI. The trough of disillusionment for AI-assisted development has begun. Byron Cook, VP and Distinguished Scientist at Amazon, founder of AWS's Automated Reasoning Group (300+ scientists, 15+ teams), says it plainly: "Generative AI is sliding into the trough of disillusionment." The headlines are shifting. The "summer of vibe coding" is over. The disillusionment isn't caused by AI being useless. AI-assisted coding delivers real productivity gains. The disillusionment is caused by false assumptions about WHERE the gains come from and WHAT changes when generation gets fast. Teams expected 10x engineering. They got 10x code generation and 1x everything else. The gap between expectation and reality is the trough. This series names the eight assumptions, explains why each one fails, and presents the resolution — not from theory, but from domains that hit the same wall and climbed out. The Eight Fallacies 1. Faster code generation means faster engineering. You made one sub-system 10x faster. Seven others didn't change. The system doesn't get faster — it breaks at the interfaces. The CPU-memory wall tells you exactly what happens and what fixes it. 2. If the output looks correct, it is correct. AI-generated code is optimized for plausibility, not correctness. It compiles, passes tests, and reads well — while violating properties nobody tested. Plausible is not correct. The gap is where production failures live. 3. You can verify AI output with another AI. Guardrails, LLM-as-judge, AI code review — the verifier has the same failure modes as the thing

Platform team sprint planning in 2026 begins with budget allocation, not architecture review. The first question is no longer "what do we need to build?" — it's "what can we afford to run?" This is not FinOps adoption. This is authority displacement. The platform team became a finance team because the control plane for infrastructure decisions migrated from architecture governance to budget governance. Cost constraints don't inform architectural decisions anymore — they dictate them. And when financial systems gain veto authority over technical systems, resilience becomes the variable that adjusts. Platform team cost governance is now the primary control surface. Architecture is secondary. How We Got Here The timeline is sharper than most organizations admit. 2018–2022 was the cloud adoption phase. Platform teams built for scale. Multi-region resilience was standard. Observability was deep. Auto-scaling was elastic. Architectural requirements shaped cost models. The budget followed the design. 2023–2024 brought FinOps as a cost visibility layer. Teams could finally see where money was going. Dashboards got built. Anomaly detection got configured. Attribution models got refined. But visibility was still separate from authority. The FinOps team reported. The platform team decided. 2025–2026 is when cost governance moved from reporting to gating. The turning point: platform teams stopped asking "can we build this?" and started asking "can we afford this?" Engineering roadmaps became cost roadmaps. Feature requests now come with budget allocation approvals. Architecture reviews now include CFO sign-off gates. This shift introduced Budget-Normalized Architecture — systems designed around predictable monthly spend targets instead of operational resilience targets. The architecture no longer optimizes for failure domains, latency requirements, or recovery objectives. It optimizes for staying under the cost ceiling. Cost governance expanded because engineering governance fa

n fan-out microservice architectures, slow-but-completing requests accumulate across services and drive p99 latency far higher than per-service metrics suggest. This article presents an adaptive hedging mechanism that uses DDSketch for real-time quantile estimation, windowed rotation to handle distribution drift, and a token-bucket budget to prevent load amplification. By Prathamesh Bhope

The conversation about cognitive debt in AI-assisted development has been framed as a tradeoff: you can go fast, or you can understand your system, but not both. The proposed mitigations — pair programming, code reviews, requiring a human to understand each change — are braking mechanisms. They trade speed for comprehension. TRIZ (Theory of Inventive Problem Solving) says braking is a compromise, not a resolution. A resolved contradiction eliminates the conflict. You don't choose between speed and understanding. You restructure the system so they don't conflict. There are six root causes of cognitive debt in AI-augmented development. Each one is a contradiction. Each one has a TRIZ resolution that doesn't involve slowing down. Root Cause 1: The Velocity-Comprehension Gap AI generates complex logic in seconds that would take a human hours to write. The human never spends the time typing the code during creation. The theory of the program is never fully formed. The Contradiction Technical contradiction: Improving development speed (AI generates code faster) worsens depth of understanding (human doesn't internalize the logic). Physical contradiction: The development process must be simultaneously FAST (to capture AI's productivity gains) and SLOW (to allow human assimilation of the system's behavior). Resolution: Separation in Space (Principle 2 — Extraction + Principle 1 — Segmentation) The contradiction assumes that the thing being understood IS the code. Extract the understanding target from the code and put it somewhere else — a smaller, slower-moving, human-readable artifact that captures what the code must satisfy, not how it works. Segment the system's theory into independent, composable units. Each unit is one property: "this service must never accept unauthenticated requests," "this data pipeline must preserve ordering," "this retry loop must terminate within 30 seconds." Each property is 1-3 sentences in natural language or 3-10 lines in a predicate language.

In part 1 , the single-document case was easy. In part 2 , two documents brought Write Skew, and we saw that even a native ACID transaction — snapshot isolation — lets it through. So teams reach for the reflex fix: a distributed lock — Redis-based, often a Redlock-style implementation. Acquire a lock on a key, do your Read → Modify → Write, release. On paper, you've finally serialized the critical section — operationally, at least. In practice, you've stepped on three mines. 1. Network latency Every guarded transaction now makes extra round-trips to Redis — before and after hitting your NoSQL store. You've doubled your coordination surface and taken a hard dependency on a second system being up, reachable, and fast on the hot path of every write. The "fast" database is now gated by the lock service. And the coupling bites harder than the average latency suggests: every Redis tail-latency spike becomes your write-latency spike — your p99 inherits Redis's p99 — and if Redis fails over mid-transaction, the lock you think you're holding can effectively vanish on the new primary, dropping you straight into the corruption case below. 2. Deadlock You can dodge deadlock entirely with a single coarse lock — but then every writer serializes on it, and you've thrown away the very concurrency you reached for NoSQL to get. So to keep throughput you go fine-grained, one lock per resource — and the moment an invariant touches more than one key (across this series, it always does), deadlock is back on the table: Transaction A locks key X, then needs Y. Transaction B locks Y, then needs X. Both block until timeout or intervention. The textbook cure — real deadlock detection, maintaining a wait-for graph across every lock holder and breaking cycles as they form — is a distributed-systems project in its own right: not something you bolt onto a cache you reached for precisely to save engineering time. So nobody builds it. Instead teams impose a standing discipline: always acquire locks

The Problem We Were Actually Solving During load testing at 50k concurrent hunters hitting the hunt endpoints, p99 latencies stayed under 200ms. But at 270k concurrent users in production, the hunt page suddenly took 1.8 seconds to load, triggering cascading 502s from our CDN. The error surfaced in Datadog as hunt_page_render_time_bucket{le=2.0} = 42% while le=0.5 dropped to 18%. The fingerprints were identical across three regions: high latency correlated exactly with Redis cache miss rate spiking from 12% to 48% during the hunt start window. Our cache-aside pattern with a 30-second TTL was amplifying miss storms. We discovered that the treasure hunt start time was synchronised by marketing campaigns. When the clock struck 10:00:00 UTC, 270k users hit the endpoint within 30 seconds. Each request would check the cache (miss), fetch from PostgreSQL, render the page, and write the cache entry. But PostgreSQL couldnt keep up with 9k queries per second during that window, causing query queueing and connection exhaustion. The Redis layer, designed for 150k ops/sec, was not the bottleneck. The database was. What We Tried First (And Why It Failed) Our first attempt was to increase Redis TTL from 30 seconds to 5 minutes. This reduced cache misses from 48% to 24%, and p99 latency improved to 650ms. But at 320k concurrent users, the latency still spiked to 1.4s because the underlying database queries were still hitting the same table with the same indexes. The Redis layer was masking symptoms, not solving the root cause. Next, we tried database read replicas. We spun up three read replicas and routed hunt queries to them using a weighted service mesh. This worked for a few minutes, but then we hit replication lag. The replicas fell 800ms behind primary, causing hunt pages to display stale treasure locations. Our operators started getting customer complaints about seeing the wrong treasure coordinates. We rolled back within 15 minutes. We even tried increasing PostgreSQL share

The Problem We Were Actually Solving Treasure hunts in Hytale arent just about generating loot. Theyre about generating simultaneous loot across thousands of players while keeping the world state consistent. We started with the assumption that events are stateless notifications: a hunt starts, we fire an event, clients react. That model worked fine when we had 200 concurrent players. At 2,000 players, the event bus turned into a 40 MB/s firehose of JSON blobs. Each loot drop required serializing the entire chunk state—blocks, entities, metadata—so clients could render the drop in real time. The JVMs G1GC couldnt handle the allocation rate. Every 47 minutes, a GC cycle would pause for 4.2 seconds, the chunk cache would fragment, and the server would hard crash with an OutOfMemoryError in net.minecraft.server.MinecraftServer#processQueue. The real problem wasnt the hunt logic. It was the architectural laziness of treating events as a catch-all glue layer instead of a boundary layer with explicit interfaces. What We Tried First (And Why It Failed) We tried Kafka as the event bus. The plan was to shard hunts by region and stream loot drops as compacted topics. The first run worked for about 6 hours before the compacted topics started to bloat. Each hunt was generating 700 KB of serialized chunk state per drop. At 30 drops per hunt per minute, thats 21 MB per hunt per minute. With 400 active hunts, the brokers couldnt keep up. The lag grew to 12 seconds, clients started rubber-banding, and we got a flood of Discord reports: You sank my boat! The event stream was now the bottleneck, not the event source. Next, we tried Redis Streams with a Lua script to aggregate loot drops per chunk. Within 30 minutes, we hit the 4 GB maxmemory limit because Lua scripts were stacking dropped items in memory while waiting for the next batch. The script was elegant—O(1) per drop—but the memory footprint made it unusable in production. Finally, we tried a sidecar service: a small Go process

Why Does Using an ORM Decrease Database Performance? While trying to optimize the shipping module in a production ERP, I noticed that database queries were incredibly slow. At first, I examined the SQL queries and checked the indexes. However, I couldn't get the performance boost I expected. The problem lay in the Object-Relational Mapper (ORM) library, which was the cornerstone of our application. ORMs make things easier for software developers by providing an abstract layer for database operations, but this convenience often comes at a performance cost. In this post, I will explain why using an ORM decreases database performance, using concrete examples from my own field experience. The core promise of ORMs is to keep developers away from SQL and allow them to interact with databases in a way that is more aligned with the object-oriented paradigm. This is a huge advantage, especially in small and medium-sized projects or rapid prototyping processes. However, when things get complex and performance becomes critical, the efficiency of the queries generated by ORMs starts to be questioned. In many cases I have encountered, especially in enterprise software development processes, the default behaviors of ORMs created an unexpected load on database servers. Query Inefficiencies Generated by ORMs ORMs usually manage database relationships using mechanisms like "eager loading" or "lazy loading". Depending on the developer's preference, these mechanisms either fetch all related data at once (eager loading) or fetch it in pieces as needed (lazy loading). However, ORMs may not always perform these loads in the most optimized way. For example, while only a few fields like ID and name are sufficient in a list view, the ORM might query the entire table or all related tables. This situation causes unnecessary data transfer and unnecessarily overloads the database server. To give an example, on the order list screen of an e-commerce site, we needed to display the customer inform

Microsoft added sandboxed code interpreters to Azure Logic Apps, enabling agents within integration workflows to generate and execute Python, JavaScript, C#, and PowerShell in Hyper-V isolated sessions. Architects get full control over model selection per workflow. The capability positions Logic Apps as an agent platform for integration alongside Foundry and Copilot Studio. By Steef-Jan Wiggers

InfoQ expands its online certification portfolio with new AI Engineering and Organizational Architecture cohorts, giving senior practitioners a confidential peer group to pressure-test production AI, platform, team design, and architecture decisions. By Artenisa Chatziou