标签：#systems

While mass-producing web tools with an AI, I've changed how I lock the design in three stages. The previous post I wrote about that got this comment: "I'd like to see the JSON approach and the design-system approach side by side." Taken at face value, I should just put the two side by side. But first, let me add a short preface. I don't want to frame this as "the JSON approach versus the design-system approach." When I called the JSON approach a "failure" in that post, I didn't mean the method is inferior; I meant it didn't suit my particular set of tools. A page made with the JSON approach does look thin. But where that thinness comes from is easily misread. Whether the design drifts and whether it looks rich are decided separately. What stops the drift is locking the design; whether it looks rich is how much you build out. What locking with JSON removes is drift in the items you specified in the schema. Whether the screen becomes rich, on the other hand, is determined by how much you've built out the machinery that turns that JSON into a screen. So it isn't that locking with JSON is what made it look like a spreadsheet. In the previous post, too, I wrote that fattening the schema and the renderer does increase the expression itself. But that came with a caveat: past a point, it heads toward rebuilding HTML and CSS by hand. What I really want to check is one step past that. If the template sets the ceiling on expression, then building out the JSON side's template as much as the current one should produce the same screen. So what does that build-out demand? I actually built it and measured. I'll share the result, along with the JSON-approach and design-system-approach screens placed side by side under matched test conditions. I'll admit up front: at the time, I chose the design system without running this comparison. So this is me building the road I didn't take, after the fact, and measuring what that cost consists of. Same order, same one-shot So that the comparis

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry . You Know What's Worse Than Not Being Able to Log In? Being told everything worked right up until you try to actually use your account. Yes, that was a real bug. And, somehow, I ended up being pulled into another authentication mystery. At this point, I’m starting to think authentication bugs have a personal grudge against me. 😅 In my previous Smash Story , I wrote about a bug where users simply couldn't log in. This time, the problem was sneakier because most of the flow looked completely healthy. The user was approved, the background task ran, the email and SMS arrived, and Cognito had a user. Then the user actually tried to use their account. And everything fell apart. It Started With Two User Pools The authentication setup was fairly large and had evolved over time, so there wasn't one shiny User Pool doing everything. We had an older Cognito User Pool supporting existing authentication flows, including mobile-based signup, while a newer User Pool handled a newer flow where users received an email containing their PIN. Both pools were intentional because they supported different parts of the authentication journey. That wasn't the problem. The interesting part was that the application database had its own representation of a user, while Cognito had another. On top of that, some of the work connecting those two systems happened asynchronously. As long as everyone agreed about who the user was, nobody cared. The moment they disagreed, authentication became very interested. The Tiny Timing Window The problem appeared in the partner and dependant journey. A member could create a partner or dependant during signup or later from the member details area. A relevant non-member user would then approve the account, which scheduled an asynchronous task called SendingEmailsAfterApprovalBot in a TaskList database table. That task ran every 15 minutes, and once it executed, the partner or depend

Welcome to Part 1 of the Go Distributed Systems Lab series! Over the course of 20 hands-on projects, we are building core distributed systems primitives from the ground up using Go 1.22+ and the standard library ( net , sync , context , log/slog , encoding/binary ). Before jumping into raw socket framing, gossip protocols, or Raft consensus, we need to master the foundational concurrency building blocks inside a single process: Goroutines, Channels, and Communicating Sequential Processes (CSP) . 💡 The Philosophy: Share Memory by Communicating In traditional concurrent programming (like C++ or Java), thread synchronization often relies on shared memory protected by mutexes, lock-free queues, or read-write locks. Go flips this model with a core design principle: "Do not communicate by sharing memory; instead, share memory by communicating." By passing ownership of data structures through Go channels, each pipeline stage operates on isolated memory. This eliminates data races by design without requiring explicit lock management ( sync.Mutex ). 🏗️ Architecture & Component Design In this first module ( 01-message-passing ), we construct a 3-stage data processing pipeline: +------------------+ Job Channel +------------------+ Result Channel +-------------------+ | Producer | -------------------------> | Worker | ------------------------> | Collector | | (Generates Jobs) | (Buffered, cap=10) | (Isolated State) | (Buffered, cap=10) | (Aggregates Data) | +------------------+ +------------------+ +-------------------+ 1. Ingestion Stage (Producer) Generates typed Job values and pushes them into a direction-constrained buffered channel ( chan<- Job ). When generation finishes, it closes the channel to broadcast an end-of-stream signal. 2. Processing Stage (Worker) Consumes from <-chan Job using Go's for job := range in construct. The worker maintains internal execution metrics (e.g., processedCount ) entirely within its local stack scope—no locks required. 3. Collector Stage R

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!) View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler What happens when thousands of concurrent requests compete for a small pool of reusable resources? You can put a mutex around a slice and hope for the best. Or you can design the scheduler around concurrency from the beginning. I chose the second option. I built Concurrent Resource Scheduler (CRS) , a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load. It was designed from the ground up for production readiness. The core library supports Go 1.22+ and is intentionally built with zero third-party dependencies . Extended features like Prometheus telemetry are strictly separated into an optional nested Go module ( Go 1.25+ ) to keep the core scheduler dependency graph perfectly empty. The core idea is simple: MANY CONCURRENT REQUESTS │ ▼ ┌───────────────────┐ │ Resource Scheduler│ └─────────┬─────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ Priority Acquire State Heap Strategy Management │ │ │ └──────────────┼──────────────┘ │ ▼ BEST AVAILABLE RESOURCE But making that work correctly under concurrency is where things get interesting. CRS is designed for use cases such as: LLM/API gateways API key pools proxy rotation database replicas GPU workers backend pools worker resources connection pools rate-limited providers reusable compute resources The scheduler itself does not know what a resource means. It only knows: "I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller." Table of Contents The Problem The Naive Approach Why a Global Mutex Becomes a Problem The Core Idea Behind CRS Architecture at a Glance Sharded Priority Heaps Why Sharding Helps The O(1) Lookup Map Priority and Acquire Are Different Problems Acquire Strategies Round Robin Weighted A

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!) View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler What happens when thousands of concurrent requests compete for a small pool of reusable resources? You can put a mutex around a slice and hope for the best. Or you can design the scheduler around concurrency from the beginning. I chose the second option. I built Concurrent Resource Scheduler (CRS) , a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load. The core idea is simple: MANY CONCURRENT REQUESTS │ ▼ ┌───────────────────┐ │ Resource Scheduler│ └─────────┬─────────┘ │ ┌──────────────┼──────────────┐ │ │ │ ▼ ▼ ▼ Priority Acquire State Heap Strategy Management │ │ │ └──────────────┼──────────────┘ │ ▼ BEST AVAILABLE RESOURCE But making that work correctly under concurrency is where things get interesting. CRS is designed for use cases such as: LLM/API gateways API key pools proxy rotation database replicas GPU workers backend pools worker resources connection pools rate-limited providers reusable compute resources The scheduler itself does not know what a resource means. It only knows: "I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller." Table of Contents The Problem The Naive Approach Why a Global Mutex Becomes a Problem The Core Idea Behind CRS Architecture at a Glance Sharded Priority Heaps Why Sharding Helps The O(1) Lookup Map Priority and Acquire Are Different Problems Acquire Strategies Round Robin Weighted Acquire Adaptive Acquire Affinity Routing Shared vs Exclusive Acquisition Resource Lifecycle Atomic State Transitions The Inactive Store Batch Operations Updates Without Destroying Heap Ordering Cooldowns Asynchronous Events Observability Prometheus Integration Concurrency Model Complexity Testing the Library Race Detector Validation

Netflix has described how it redesigned the streaming pipeline behind Service Topology, its real-time service dependencies map, to support production scale. The system uses three stages to separate intermediary resolution from enrichment and persistence, propagates backpressure to Kafka rather than dropping records, and uses server-sent events instead of gRPC for high-volume internal transfers. By Eran Stiller

Every API that sits behind an unreliable network eventually faces the same problem: a client sends a request, the connection drops before the response arrives, and the client has no idea whether the operation happened. Did the payment go through? Did the order get created twice? The client's only safe move is to retry — which means your server needs a story for what happens when the same "create this thing" request arrives more than once. That story is idempotency keys, and getting the details right is more subtle than it first looks. The core idea The client generates a unique token — typically a UUID — once per logical operation, and attaches it to every retry of that operation: POST /orders Idempotency-Key: 7c3fd9a2-df01-4b3e-9a55-1e5f9b6b6d55 {"sku": "WIDGET-1", "qty": 2} The server's job is to guarantee that no matter how many times a request with that key arrives, the side effect (charging a card, creating an order, sending an email) happens at most once, and every retry gets back the same response the original request would have produced. Note what this is not: it is not deduplicating by request body. Two requests with identical bodies but no key are legitimately two different orders for two widgets. The key is what marks them as "the same attempt," not the payload. The naive approach, and why it breaks A common first pass is a table like: CREATE TABLE idempotency_keys ( key TEXT PRIMARY KEY , response_body JSONB , status_code INT ); On each request: check if the key exists, and if so return the cached response; otherwise do the work and insert the result. This looks right and is wrong in a specific way: it has a race condition. Two retries can arrive concurrently (a client that timed out and fired a second attempt while the first was still in flight), both miss the cache check, and both execute the underlying operation. You've now charged the card twice. Making the check-and-do atomic The fix is to claim the key before doing the work, using the database's ow

You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling. Then the request crosses into async land, and all of that evaporates. This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself. The flow everyone believes is fine The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller's identity downstream as headers: // api-gateway — IdentityPropagationFilter (@Order(2), after security) IdentityContext identity = identityContextExtractor . extract ( jwt ); // Always set all three headers (empty when absent) to mask any spoofed values. enrichedRequest . putHeader ( IdentityHeaders . USER_NAME , nullToEmpty ( identity . username ())); enrichedRequest . putHeader ( IdentityHeaders . USER_EMAIL , nullToEmpty ( identity . email ())); enrichedRequest . putHeader ( IdentityHeaders . USER_ROLES , identity . rolesAsString ( DELIM )); One detail here matters more than it looks. The headers are always overwritten , even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you've built an impersonation API. So far, so good. The synchronous hop carries identity. The problem starts one line later. The hidden f

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete . "The agent traces everything" is the sentence repeated in every onboarding doc. This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see . The flow everyone trusts The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic , so it would always reflect what the system actually does. Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans. The edge that wasn't there I generated the graph and the synchronous edges lit up immediately: api-gateway → user-service user-service → MySQL notification-service → PostgreSQL Then I looked for the one edge I actually cared about — user-service → notification-service , the asynchronous hop over Kafka. It wasn't there. The naive conclusion (and why it's wrong) The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer. So I checked. And the consumer was completely fine. notification

I asked Linux for one 4 KiB page from the start of a cold file. Four pages came back. I moved the same read one page further in, ran it again, and got one. Same file, same syscall, same kernel. The only thing that changed was where I started reading, and I spent twenty minutes assuming the tool I'd just written was miscounting. It wasn't. A read that starts at byte zero is treated as a promise. There's a branch in mm/readahead.c that reads, in full, if (!index) goto initial_readahead; . Offset zero means the kernel takes you for a program that's about to stream the whole file, and it fetches ahead immediately. Start anywhere else and you're assumed to be seeking randomly until a pattern proves otherwise. Nothing in my call said a word about my intentions. It inferred them from an offset. I spent two weeks on this sort of thing recently. Not for work, and not toward anything shippable. The short version of what I found is that a surprising amount of the machinery under a running program isn't carrying out instructions at all. It's guessing. The bench , because it changes how you should read every number here: an ext4 filesystem on a loop device, inside an OrbStack Linux VM on an Apple Silicon Mac, kernel 7.0.14, 4 KiB pages, read_ahead_kb at 128. That's a container sharing the host's kernel, not bare metal, and the host reclaims memory aggressively enough that a fully cached file can go cold in fifteen seconds. Reads came from dd ; the page-by-page counting came from a small C tool I wrote that mmap s a file and asks mincore() which of its pages are resident. You're not addressing the disk, you're addressing the page cache The model most of us carry is that read() goes and gets bytes off a device. It doesn't. It copies bytes out of the page cache into your buffer, and the page cache is just RAM the kernel uses to remember parts of files. If what you want is already there, no device is involved. If it isn't, the kernel fills the cache first and then copies. Either way

When engineers discuss quantum computing, the conversation usually focuses on future supercomputers cracking traditional encryption passwords in a matter of seconds. As a systems architect who spends my days building distributed platforms, which are networks of independent cloud servers working together as a single application, I see a different, highly practical challenge taking shape. The transition to quantum-resistant security is not simply a theoretical math problem. It is an infrastructure challenge that will directly impact network throughput, memory usage, and messaging efficiency across global cloud environments. To protect sensitive enterprise records and business platforms against future quantum threats, security organizations are transitioning to Post-Quantum Cryptography. This field involves building new mathematical algorithms that quantum computers cannot easily solve. However, these stronger defense mechanisms come with a major trade-off in size. Traditional cryptographic signatures, which are digital verification stamps used to prove that a data message comes from an authentic sender and was not altered, are remarkably small. An older, standard signature might only take up sixty bytes of memory. By comparison, a quantum-safe signature can easily require several thousand bytes. In a simple website, adding a few extra kilobytes to a security header goes unnoticed. But modern cloud infrastructure relies heavily on event-driven architecture, a design strategy where dozens of microservices communicate by constantly publishing tiny, real-time updates to shared message queues. In these systems, the actual business payload might only be a small status change containing twenty bytes of text. If the quantum security stamp attached to that message is three thousand bytes, the overhead of the security layer completely outweighs the actual data being sent. When security footprints expand by orders of magnitude, the physical realities of computer networking take

Cloudflare recently introduced Meerkat, an internal globally consistent control-plane service based on the QuePaxa consensus algorithm. Unlike Raft, it allows leaderless writes while preserving strong consistency, improving availability across Cloudflare's global network. By Renato Losio

This article was originally published on BuildZn . Everyone talks about multi-agent systems but few show you how to actually coordinate them without a ton of boilerplate and deadlocks. I spent weeks trying to get agents to talk, especially when building something like FarahGPT's multi-agent trading system, often hitting insane latency. Turns out, qm can drastically simplify this, and this qm multiplayer AI agent tutorial will show you how to cut task completion times by 20% using a specific Node.js workflow. Why Multi-Agent Systems Aren't Just Hype Anymore (and qm Helps) Single LLM calls hit a wall, fast. You get generic answers, struggle with complex, multi-step tasks, and prompt engineering becomes a full-time job. I've built 9-agent YouTube automation pipelines and an AI gold trading system that needed to analyze market data, news sentiment, and historical trends concurrently. Trying to jam all that into one prompt for a single agent? Forget about it. You need a collaborative AI agent architecture . That's where multi-agent systems shine. You break down complex problems into smaller, manageable tasks, assign them to specialized agents, and have them work together. Think of it like a dev team: one person focuses on backend, another on frontend, another on CI/CD. This is how you handle real-world complexity, and it's how I scaled FarahGPT to 5,100+ users. The challenge? Orchestration. How do these agents communicate? Who manages their state? How do you ensure they don't step on each other's toes or get stuck waiting for slow upstream tasks? This is exactly where qm , a lightweight agent harness, becomes a game-changer for building AI teams. It gives you the primitives to define agents, tasks, and workflows without drowning in custom event loops. The Core Concept: Task Delegation in qm Most qm examples show simple agent interactions. Agent A asks Agent B. Done. But what if Agent A needs to delegate a task that itself needs parallel sub-tasks, and then aggregate the

There are fresh signs that fusion power startup Commonwealth Fusion Systems will list in the next two to three years.

Commonwealth Fusion Systems raised $1 billion as the startup moves toward its first commercial fusion power plant.

Murat Demirbas discusses the shift toward disaggregated cloud database architectures driven by cloud economics. He explains how decoupling compute from storage enables elastic scaling, cost efficiency, and fault isolation. He shares how classical Paxos roles foreshadowed disaggregation, while analyzing network tradeoffs, shared-memory evolution, and self-assembling database designs. By Murat Demirbas

Handoffs are useful when a specialist agent needs to take over a task. They also make cost easier to hide, because the bill is spread across graph nodes instead of one visible chat turn. Why can LangGraph handoffs multiply tokens? LangGraph handoffs can multiply tokens because each model-calling node may resend instructions, prior messages, retrieved material, tool returns, summaries, and artifacts, then loops or handoffs repeat that payload for the next agent. Token amplification is the total prompt-plus-completion tokens across a trace divided by a simpler baseline for the same task; Anthropic reported in June 2025 that multi-agent systems used about 15x more tokens than chats while improving an internal research evaluation by 90.2% . Quick Answer: Handoffs raise the token bill when each agent receives copied context instead of a narrow task packet. Anthropic’s June 2025 research system showed the tradeoff clearly: multi-agent runs used about 15x more tokens than chats while scoring 90.2% higher on its internal research evaluation . In LangGraph, the practical issue is observability and budgeting, not whether graphs are bad. The LangGraph project describes the runtime as a way to build stateful, long-running agents with persistence, human control, memory, and debugging support; those same traits make it possible to measure where context grows instead of guessing. "Multi-agent systems are often highly effective at open-ended research tasks, but token usage can be substantial," — Anthropic engineering team at Anthropic The small verified demo below shows the arithmetic behind a 15x bill: a 100-token task becomes 1,500 billed tokens when 5 agents each receive 3 copies of the relevant context . """ Tiny token-accounting demo: handoffs multiply the same task context. """ task_tokens = 100 agents = 5 context_copies_per_handoff = 3 # instructions + task + summary/history direct_bill = task_tokens handoff_bill = task_tokens * agents * context_copies_per_handoff print ( f

Encrypted data stored in cloud archives today will outlive the mathematical algorithms guarding it. In enterprise architectures that handle long-term records, like construction risk logs or employee compliance platforms, data retention schedules often span twenty to thirty years. When building cloud pipelines that move this information across services, we depend heavily on asymmetric encryption, which is a security method using one public key to lock data and a separate private key to unlock it. Standard public-key algorithms rely on mathematical problems that are nearly impossible for classical computers to solve within a reasonable human timeframe. Quantum computing changes this equation entirely. Quantum computers leverage quantum mechanics, the physical rules governing subatomic particles, to perform calculations at speeds fundamentally unimaginable with traditional silicon processors. While powerful quantum systems are still in development, the security threat to distributed systems exists today. Hostile actors do not need to crack modern security algorithms in real time. Through a pattern known as Harvest Now, Decrypt Later, adversaries can capture and store encrypted network traffic right now. They simply wait until future quantum hardware becomes capable of running the formulas required to decrypt that stolen history. For software architects, preparing for post-quantum cryptography, which refers to new mathematical encryption algorithms designed to withstand quantum attacks, is far more than a simple library swap. It is a deep distributed systems migration challenge. The primary operational hurdle is payload size and computational overhead. Quantum-resistant algorithms require significantly larger digital keys and payload headers than the standards we rely on today. When cryptographic payloads expand, every component of a distributed platform feels the ripple effect. Message queues experience higher bandwidth demands. Database indexes inflate. Memory consump

Herodotus wrote of a bird that lived five hundred years in Arabia, and when its life came to an end, it did not wait to be surprised by death. It built its own nest of cinnamon and myrrh, set the nest and itself alight, and let a new bird rise from what the fire left behind. The Hestia argued for tending a fire that must never go out. That’s true, and it isn’t the whole truth. Teams end. People leave. Companies get acquired, reorganized, shut down, and five years from now some part of this whole model will probably look as dated as the practices it was written to replace. No amount of tending prevents that. Pretending otherwise is its own kind of Alexandria , a slow decline dressed up as continuity, right up until the fire goes out anyway and nobody chose the moment. The bird in Herodotus doesn’t get caught by surprise. It builds the pyre itself. Chooses the moment, gathers what matters, and burns deliberately, trusting that what rises afterward carries the shape of what came before, not because the fire preserved the old bird whole, but because starting over was never the same thing as starting from nothing. That’s the part tending alone can’t promise. A team that’s about to be split up can hand its shared model to whoever inherits the work on purpose, the way a rep in the Boule carries a decision back instead of leaving it to travel however it happens to travel. A team about to lose its most experienced person can spend the weeks before that departure making sure the framing, not just the conclusions, made it into someone else’s head, the way the Mimesis argued a junior actually learns. None of that stops the ending. It decides what the ending leaves behind. This series doesn’t get to end with a fire that never goes out. Nothing does. It gets to end with the only thing actually inside anyone’s control. Build the pyre on purpose. Choose what goes into the fire. References The Myth of the Phoenix: Rebirth and Renewal : Greek Mythology, on Herodotus’s original accoun

Part 1 documented the recurring snapback in practice. This note asks a narrower question: what does the observed pattern support, what remains a working hypothesis, and what changes should follow in the project? Status: Bounded project conclusion. This note separates observed behaviour, working hypothesis, and practical consequence. It is based on current project documents and interactions; it is not external validation or a universal claim about AI systems. What the evidence supports 1. The project already contains a stable relational model The working model is not generic “AI assistance.” It separates reasoning surfaces, uses bounded comparisons, permits two-way cognitive pressure, and keeps final acceptance authority with the human. Reciprocal cognitive contribution, asymmetrical governing authority. 2. Concrete project work preserves the structure better than public abstraction At the concrete level, instructions such as: Review this proposal against that architecture. preserve the distinction between the object being reviewed, the surface applying pressure, the evidence, and the authority that may accept a change. When the same structure was compressed into general prose, generated explanations repeatedly returned to a simpler one-way model of either human control or transferred AI authority. That is an observed pattern in this development process. 3. Public explanation is a separate reasoning surface A README, article, summary, or portfolio page is a projection of the model, not the model itself. It cannot be assumed to reproduce the internal structure faithfully merely because that structure is present in context. The explanation must be reviewed against the model it represents: Does this explanation preserve the actual authority, review, evidence, and state-transition structure? 4. Annoyance was useful boundary data The irritation indicated that the generic rendering was no longer merely an imperfect exploration. It was colliding with an internal frame that