标签：#systems

I was testing out CH-Ops - an admin GUI for self-hosted ClickHouse - on a simple setup: 1 shard, 2 replicas. Stumbled onto the replication queue almost by accident. Here's what I did: I stopped one of the nodes (let's call it Node B), then inserted some data through the other one (Node A). Just wanted to see what would happen. Then, while Node B was still down, I checked it in CH-Ops. It had stuff sitting in its replication queue. My first assumption was: okay, this must be showing what's left to replicate across the cluster - the total pending replication work. So I switched over and checked Node A, the one that was actually up and had just received the insert. Its queue was empty. That didn't match what I expected at all. If the queue was a cluster-wide "here's what still needs to replicate" view, Node A should've shown something too - it was the one that had the fresh data now waiting to reach Node B. Instead it was Node B, the down one, sitting there with pending tasks. That mismatch is what sent me digging. Turns out the queue isn't cluster-wide at all - it's specific to each ClickHouse instance. Once I brought Node B back up, its queue drained in seconds and the data showed up. That whole experiment is basically the entire post in miniature. Here's the mental model I ended up with. A Queue Belongs to a Replica, Not to the Table This is the first thing to get straight. With a ReplicatedMergeTree table, you can have multiple replicas holding copies of the same data. It's tempting to think of replication as one shared pipe between them. It isn't. Each replica keeps its own local replication queue . So if you see: Replica 1 → queue_size = 0 Replica 2 → queue_size = 25 that doesn't mean 25 operations are waiting somewhere in the middle for both replicas to pick up. It means Replica 2, specifically, has 25 tasks it hasn't finished yet. Once that clicked for me, the rest of the system made a lot more sense. So Where Do These Tasks Come From? Replication in ClickHouse

Much of the traditional software operations we deal with are online, we click a button, wait for a moment, and the transaction or operation is completed. But there is a big area that deals with software operations that require offline processing. For example, background processing of jobs, e.g., OpenAI training/improving its existing GPT models behind the scenes using the data it gathers from its users. Batch Processing Whenever such an offline system runs a job that typically generates output from a batch of inputs, we call that batch processing. Inputs here are immutable, which avoids side effects. Benefits of batch processing: You can time travel. In case of any failure or unintentional outputs, you can jump to the last input checkpoint before a batch processing job. This handling is often referred to as human fault tolerance. Using batch processing and offline systems, compute usage efficiency can be improved. For example, whenever a heavy computation needs to be done, it's better to do it in bulk on maybe a GPU compute rather than crashing the CPU host where the server is online. Though the boundary between online and batch processing is not always clear. For example, a long-running database query could also be categorised as batch processing. Another alternative to batch processing is stream processing, which we will understand in the next article. MapReduce MapReduce is a batch processing algorithm that is utilized by Hadoop, CouchDB, and MongoDB as well. It is a balanced approach that is less extreme than completely parallelizing the jobs. There are several other frameworks like this that are now replacing MapReduce. For example, DataFrames APIs, query languages, etc. We will see MapReduce in detail sometime later. Simulating Batch Processing with Unix Tools (Single Host) If you are a Linux user, this simulation could be very easy for you to grasp. If not, just put it in ChatGPT or any AI tool to understand the command in detail if interested. A typical Ngin

The Transactional Outbox Pattern: Dual-Write Consistency in Distributed Systems One of the most dangerous anti-patterns in microservices architecture is the Dual-Write Vulnerability : updating a database record and immediately publishing an event to a message broker (e.g., RabbitMQ, Kafka) in the same API call. If the network fails or the broker is unavailable after the database transaction commits, the event is lost forever. Conversely, if the event publishes but the database rollback triggers, downstream consumers process a phantom event that does not exist in the source of truth. In this deep dive, we architect the Transactional Outbox Pattern with Change Data Capture (CDC) to guarantee At-Least-Once delivery with zero distributed locking overhead. Technical & Interview Cheat Sheet Approach Consistency Guarantee Failure Mode Overhead Dual Write (Naive) None (Eventual inconsistency) Message lost if broker drops Low 2-Phase Commit (2PC / XA) Strict Atomicity Blocking locks, single point of failure Very High Transactional Outbox (Polling) At-Least-Once Polling query table contention Moderate Outbox + CDC (Debezium) At-Least-Once (Zero Table Locking) Requires WAL decoder plugin Optimal 1: Database Schema Design The business entity change and the outbox event MUST commit within the exact same database transaction: -- Business Entity CREATE TABLE orders ( id UUID PRIMARY KEY DEFAULT gen_random_uuid (), customer_id UUID NOT NULL , total_amount NUMERIC ( 12 , 2 ) NOT NULL , status VARCHAR ( 32 ) NOT NULL , created_at TIMESTAMPTZ NOT NULL DEFAULT NOW () ); -- Transactional Outbox Table CREATE TABLE outbox_events ( id UUID PRIMARY KEY DEFAULT gen_random_uuid (), aggregate_type VARCHAR ( 64 ) NOT NULL , aggregate_id VARCHAR ( 64 ) NOT NULL , event_type VARCHAR ( 64 ) NOT NULL , payload JSONB NOT NULL , created_at TIMESTAMPTZ NOT NULL DEFAULT NOW () ); -- Index for high-throughput CDC streaming CREATE INDEX idx_outbox_created ON outbox_events ( created_at ); 2: Atomic C# Tra

Airbnb redesigned its authentication architecture around server driven flows and policy based challenge selection. The new Flexible Authentication system reduced authentication related code by 60%, cut the web client bundle by 100 KB, improved successful authentication by 2.6%, reduced duplicate account creation by 27%, and lowered OTP costs by 11%. By Leela Kumili

In traditional software architectures, we almost instinctively reach for the CRUD (Create, Read, Update, Delete) paradigm. We design an entity model, map it to a relational schema using an ORM, and use that identical abstraction to both alter state and display data on user dashboards. For simple applications, this works flawlessly. But as systems scale—both in business complexity and throughput, this dual-purpose model starts showing fractures: Write logic demands tight validation, transactional boundaries, normalization, and domain invariants. Read logic demands flat, pre-aggregated, denormalized representations across dozens of tables to serve responsive UIs. Trying to satisfy both masters with a single schema leads to unwieldy SQL joins, lock contention, compromised domain boundaries, and performance gridlock. This is where Command Query Responsibility Segregation (CQRS) enters the picture. 1. What is CQRS? Coined by Greg Young and based on Bertrand Meyer’s Command-Query Separation (CQS) principle, CQRS states that an application should use separate models to update and read data . At its philosophical core: Command (Write): Represents an intent to alter domain state (e.g., SubmitOrder , DeactivateUser , ChangeBillingAddress ). A command should focus entirely on domain logic, data integrity, and business rules. In strict CQRS, commands do not return domain data — only an acknowledgment, validation failure, or generated entity ID. Query (Read): Retrieves data without mutating application state (e.g., GetOrderSummaryById , ListCustomerInvoices ). Queries should execute side-effect-free operations that return lightweight Data Transfer Objects (DTOs). ┌────────────────────────────────────────────────────────┐ │ Client │ └─────────────┬────────────────────────────▲─────────────┘ │ │ Execute Command Run Query │ │ ▼ │ ┌───────────────────────────┐ ┌───────────┴─────────────┐ │ Command Model │ │ Query Model │ │ (Validation & Invariants) │ │ (Optimized for DTOs) │ └──────

Sooner or later you hit the same small problem: two services, on two machines, want to touch the same thing at the same moment — append to a shared file, update a row nobody is fencing, call an API that tolerates one caller at a time. One of them has to wait. The usual answers feel heavier than the problem. Put a service in front and serialize everything through it — now you are building a queue, and then a second queue to hand results back, because you no longer know the outcome at the moment you asked. Cache the resource in Redis and lock there — fine until the resource does not fit in memory, and you have inherited Redlock's ordering guarantees (there are none) and its debates. I wanted the lock as its own primitive: lock a key, do the work, unlock the key. Nothing else. That is Locking-Center — a single binary, one dependency, no config file, no consensus layer to operate. This post is about the three ideas that made it small enough to be worth trusting. 1. One channel per key — and the queue comes for free Every key gets a Go channel with a buffer of exactly one: type Channel struct { key string mutexChan chan bool // buffered, capacity 1 } func NewChannel ( key string ) * Channel { return & Channel { key : key , mutexChan : make ( chan bool , 1 )} } Sending into it acquires the lock. Receiving from it releases : c . mutexChan <- true // acquire — blocks if someone already holds the key // ... critical section ... <- c . mutexChan // release The buffer of one is the whole trick. The first send fills the buffer and returns immediately: that caller holds the key. The second send has nowhere to go, so it blocks — and so does the third, and the fourth. The blocked senders are the wait queue. When the holder releases (a receive frees the slot), the runtime wakes the next blocked sender. And it wakes them in order. The Go runtime keeps a FIFO wait queue behind every channel, so callers are served roughly in arrival order rather than whoever happens to reschedule firs

Model Predictive Control for Real-Time Robot Navigation A path planner tells a robot where it should go. A controller determines how the robot should move to follow that path. Model Predictive Control (MPC) repeatedly predicts future behavior and chooses control inputs that optimize a short horizon. MPC Concept Current State | v Predict future states | v Optimize control sequence | v Apply first control | v Measure new state | +----> Repeat The key idea is that the entire control sequence is not executed at once. Only the first action is applied before the problem is solved again. Robot Model For a simple differential-drive robot: x_dot = v cos(theta) y_dot = v sin(theta) theta_dot = omega The controller can predict where the robot will be after applying candidate velocity commands. Optimization Objective A typical objective might penalize: Distance from reference path Heading error Excessive control effort Rapid control changes Collision proximity Conceptually: Cost = tracking_error + control_effort + smoothness_penalty + obstacle_penalty Prediction Horizon Suppose the controller predicts: t0 -> t1 -> t2 -> t3 -> t4 For each candidate control sequence it estimates the resulting trajectory. The optimizer selects the best feasible sequence. Obstacle Handling A cost function can strongly penalize trajectories near obstacles: Obstacle ### ##### ### \ predicted trajectories \---- safe \--- unsafe Hard constraints can also be used when collision avoidance must be guaranteed by the optimization formulation. ROS 2 Architecture /global_plan | v /mpc ^ | /odom /imu /local_costmap | v /cmd_vel Real-Time Requirements MPC is computationally heavier than simple feedback controllers. Monitor: Optimization time Control frequency Solver failures CPU utilization Prediction horizon Sensor latency If optimization misses its deadline, the system needs a safe fallback. Practical Implementation Strategy Start simple: Define a robot model. Implement trajectory prediction. Define tracking

In this episode, Scott Jenson, a veteran UX designer known for his work on the Macintosh, Google Maps, and Chrome examines the long-term stagnation of desktop operating systems and the limitations of current mobile and cloud-centric models. By Scott Jenson

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback. A couple of posts back we spent a while inside XOR distance , then used it to build Kademlia , the DHT algorithm that lets a network find anything without a directory. Kademlia: Algo That Turned XOR Distance Into a Network Athreya aka Maneshwar Athreya aka Maneshwar Athreya aka Maneshwar Follow Aug 26 Kademlia: Algo That Turned XOR Distance Into a Network # webdev # programming # beginners # algorithms 20 reactions Add Comment 6 min read I promised that algorithm shows up "under BitTorrent, IPFS, Ethereum." Today we cash that check. We're taking BitTorrent apart, piece by piece, and Kademlia is going to walk right back in through the side door. Also, fun fact before we start: a suspicious number of people on Reddit think Bram Cohen, the guy who wrote BitTorrent alone in Python in 2001, is secretly Satoshi Nakamoto. I'm not saying it's true. I'm saying that by the end of this post you'll understand why people keep saying it. The number that should not have been possible In 2004, a measurement firm called CacheLogic reported that BitTorrent alone was responsible for roughly 35% of all internet traffic. More than every other peer to peer network combined. More than the entire web. One protocol. Written by one guy. No company. No datacenter. No servers anywhere with "BitTorrent Inc" on the rack. That last part is the whole story. Every "normal" system you've ever worked on scales by throwing money at it: bigger box, more replicas, a CDN in front. BitTorrent had nobody to throw money at anything, so every hard problem, capacity, trust, scheduling, incentives, discovery, had to get solved inside the protocol itself . Problem 1: the client-server ceiling has a name Distributing a file in 2001 meant one server, one uplink, and every download eating

Kasia Trapszo discusses how Netflix evolved its commerce platform from a U.S. DVD service into global infrastructure. She explains navigating international payment realities, adapting to strict regulatory mandates, decomposing monolithic architectures along domain boundaries, and re-architecting systems for massive live-event demand - proving great systems survive by continually evolving. By Kasia Trapszo

Cloudflare recently open-sourced Cloudflare OS. It allows enterprise teams to output work artifacts grounded in enterprise knowledge, know-how, and provisioned connectors, automate repetitive workflows with optimized token cost (with AI assistance only where needed), and build personal, shareable, customizable work software that caters to specific, complex use cases within a secure sandboxed model By Bruno Couriol

VRP Is Ready for External Validation — Who Will Be the First to Pilot It? My name is Vitalijus Riabovas. I am the independent architect and creator of VRP — Veil Routing Protocol . VRP is a continuity-first networking architecture built around a simple principle: A logical session should not have to die simply because the network underneath it changed. Wi-Fi → LTE/5G. IP mutation. NAT / CGNAT churn. Temporary blackout. Path failure. Recovery. Replay attempts. Stale authority. Duplicate execution. For a long time, VRP was primarily architecture, runtime engineering and internal validation. That stage has changed. The public validation boundary exists now. And I am inviting serious engineers and organisations to test it. DON'T TRUST MY CLAIMS. TEST THEM. I am not asking the networking industry to believe a presentation. I built the measurement boundary. The public VRP Validation Kit provides engineers with an environment for evaluating observable behaviour independently. You can: clone the repository; run the Docker scenarios; inspect generated evidence; verify manifests and hashes; attack the evidence; delete events; duplicate events; reorder events; attempt replay; introduce stale conditions; corrupt artifacts; run the verifier; reproduce PASS / REJECT / INCOMPLETE outcomes. If you believe something is wrong, try to produce a reproducible contradiction. Give me: environment → scenario → commands → evidence → result That is useful engineering. WHAT HAS BEEN BUILT? VRP has moved far beyond an architectural diagram. The project now includes multiple engineering layers. Continuity architecture Logical session identity is designed to survive changes in the underlying network path. The architecture is being developed around continuity rather than assuming that transport identity and logical session identity must always be the same thing. Runtime The protected runtime implements the private VRP mechanisms. That implementation is not public . State and transition handling T

The database platform Harper advocates for a single-runtime architecture that keeps application code and data together, with its benchmark against a Vercel-based stack reporting significantly better performance on live, personalized-data workloads. Harper recently released version 5.2, with a new record cache and more throughput per node. By Renato Losio

In the previous part of this series, we built a tiny supervisor from scratch. When a worker crashed, the supervisor started a replacement. That replacement had a new PID: old worker -> #PID<0.102.0> new worker -> #PID<0.105.0> This reveals an important limitation of sharing PIDs as a public interface. A PID identifies one running incarnation of a process. It is excellent for sending a reply, setting up a monitor, or creating a link. It is not a stable address for a service that may stop and later be replaced. In this part, we'll use named processes to give a worker a discoverable address: :worker We'll build three small examples using: Process . register / 2 Process . whereis / 1 :global . register_name / 2 :global . whereis_name / 1 send / 2 No GenServer . No OTP Registry . The goal is to understand the lookup problem that registries solve before reaching for those abstractions. The PID-Sharing Problem Suppose one process starts a worker and gives its PID to a client: worker = spawn ( fn -> worker_loop () end ) send ( client , { :worker_started , worker }) The client can now send work directly: send ( worker , { :work , self (), "hello" }) This works while that particular worker process is alive. But process IDs are temporary. If the worker exits, the PID is no longer a route to the service: Client Worker holds #PID<0.102.0> #PID<0.102.0> | | | X exits | | send(#PID<0.102.0>, work) |------------------------------> no worker receives it Sending to a dead local PID does not raise an error and does not restart a process. The message is simply not delivered to a living worker. One answer is to tell every client about every new PID after a restart. That spreads lifecycle knowledge throughout the system. Another answer is to make clients depend on a name and resolve that name when sending. Registering a Local Name Our first worker waits for a stop message: defmodule Worker do def start do spawn ( fn -> receive do :stop -> :ok end end ) end end Starting it gives us a PID:

Sam Newman discusses the concept of progressive collapse in civil engineering and how it applies to distributed systems. Using real-world examples - from the 1968 Ronan Point tower failure to AWS outages - he shares crucial resilience engineering strategies for software leaders. Learn how to strengthen components, isolate failures, and reduce interconnections to prevent catastrophic cascades. By Sam Newman

I've written before about SMESH, a coordination protocol modelled on mycorrhizal networks — the fungal web that lets trees in a forest warn each other about drought and disease with nothing in charge of the network. Signals diffuse, decay on their own, and get reinforced when independently confirmed. Consensus emerges instead of being orchestrated. That was the idea. This post is about the part where I found out whether it worked. The transport that had never run SMESH has had a QUIC transport in it for a while. Roughly 500 lines: a quinn endpoint that is simultaneously server and client, self-signed certs, length-prefixed bincode frames over unidirectional streams, an accept loop that spawns per-connection and per-stream tasks, connection pooling. Every test passed. The workspace was green. I could point at smesh-runtime/src/transport.rs and say "yes, it does peer-to-peer." Then I grepped for who actually constructed it: $ grep -rn "QuicTransport" --include = '*.rs' . smesh-runtime/src/transport.rs:177:pub struct QuicTransport { smesh-runtime/src/transport.rs:192:impl QuicTransport { smesh-runtime/src/lib.rs:16:pub use transport:: { QuicTransport, ... } ; Its own definition, and a re-export. Nothing else in the workspace had ever instantiated it. No binary opened a socket. SmeshRuntime imported TransportConfig , stored it in a struct field, and never looked at it again. I had a networking layer with tests, docs, and zero executions. Three bugs in the first twenty minutes I wrote an integration test that starts two runtimes, has one dial the other, and asserts a signal crosses. Here is what fell out before it went green. 1. It panicked on the first call. Could not automatically determine the process-level CryptoProvider from Rustls crate features. rustls 0.23 refuses to pick a crypto backend when more than one is compiled in, and quinn pulls in both through its own feature set. Every call to QuicTransport::new would have panicked for anyone, ever. Nobody noticed bec

In this article, we'll explore the mechanisms to solve the coordination problem. 8. Introducing Leases To address the problem of permanent ownership, distributed systems typically replace it with temporary ownership. This concept is known as a lease . Instead of granting indefinite control over a resource, the coordination service assigns ownership for a limited period of time. Rather than stating, “You own this resource until you explicitly release it,” the system instead says, “You own this resource for the next 30 seconds.” This changes the interaction model significantly. Acquire Lease | v Execute Work | v Renew Lease | v Continue Processing As long as the application remains healthy, it periodically renews the lease to maintain ownership. If the application crashes or becomes unresponsive, it can no longer renew the lease. Once the lease duration expires, ownership is automatically revoked. At that point, another application becomes eligible to acquire the lease and continue the work. Leases solve a critical problem in distributed systems: they prevent abandoned locks from blocking progress indefinitely . The system can recover automatically without manual intervention. However, while leases improve availability, they also introduce a new class of subtle and more complex problems. Leases Depend on Time To understand the next challenge, assume the lease duration is thirty seconds. Application A successfully acquires the lease. Lease Granted Duration = 30 seconds After twenty seconds, the JVM begins a long Full Garbage Collection cycle. This pause lasts forty seconds, significantly longer than the lease duration. The timeline now becomes problematic. Lease Granted | | Processing | | GC Pause (40 sec) | | Lease Expires While Application A is paused, the lease expires. During this time, another application requests access to the same resource. The coordination service observes that the previous lease has expired and therefore grants ownership to Application B. Appl

The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap. Then you load-test it. I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned. What a correct implementation looks like Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker: @Scheduled ( fixedDelay = 5000 ) @Transactional public void publishPendingEvents () { List < OutboxEvent > batch = outboxRepository . findTop20ByStatusOrderByCreatedAtAsc ( OutboxStatus . PENDING ); for ( OutboxEvent event : batch ) { event . setStatus ( OutboxStatus . PROCESSING ); outboxRepository . save ( event ); try { kafkaTemplate . send ( event . getTopic (), event . getPayload ()). get (); event . setStatus ( OutboxStatus . PUBLISHED ); } catch ( Exception e ) { event . incrementRetryCount (); if ( event . getRetryCount () >= MAX_RETRIES ) { event . setStatus ( OutboxStatus . FAILED ); } else { event . setStatus ( OutboxStatus . PENDING ); } } outboxRepository . save ( event ); } } This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed. Here is what that correctness does not cover. Gap 1: Your throughput ceiling is a config line fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycl

A stored fact can be wrong in a quiet way. The answer still reads clean. A preference from an old exchange gets reused, the message goes out with confidence, and later nobody can tell why that detail was allowed back into the result. That is the failure I built around. When a system returns remembered material, the caller needs the text plus the reason it passed the reuse check. A log line found after the action is weak evidence. The object that leaves the memory service has to carry the admission record with it. 1. Keep the outside surface small This is the pattern I used in Holographic, Law-Bound Memory (HLM), a stand-alone memory brain outside application code. The README describes public Application Programming Interface (API) routes under /api/brain/* , with internal /api/v1/* services behind that layer. The outside shape is intentionally thin: register an agent, write a fact, build a capsule. The Python Software Development Kit (SDK) in sdks/python/hlm_sdk/client.py shows the boundary without exposing table names or policy code: import httpx class HLMClient : def __init__ ( self , base_url : str , token : str | None = None ): self . base_url = base_url . rstrip ( " / " ) self . _client = httpx . AsyncClient ( headers = { " Authorization " : f " Bearer { token } " } if token else None ) async def register_agent ( self , name : str ): r = await self . _client . post ( f " { self . base_url } /api/brain/agents/register " , json = { " name " : name }) r . raise_for_status return r . json async def write_fact ( self , text : str , tags : list [ str ] | None = None , selectors : list [ str ] | None = None ): r = await self . _client . post ( f " { self . base_url } /api/brain/memory/facts " , json = { " text " : text , " tags " : tags or [], " selectors " : selectors or []}) r . raise_for_status return r . json async def build_capsule ( self , query : str , budget_tokens : int = 2048 ): r = await self . _client . post ( f " { self . base_url } /api/brain/context/cap

TL;DR — I've been building the Social Resource Floor: an open blueprint for coordinating one person's access to basic survival resources — food, housing, energy, healthcare, and more — across many independent providers, so that reaching those resources is grounded in being human rather than in financial access. The first blueprint version is now complete: language-neutral schemas, prose specifications, a reference implementation, and a first adapter. This post is about the engineering choices behind it, and the reasons for each — how it stays a contract rather than a product, how it keeps personal data out of the coordination layer, why it binds to existing standards instead of inventing new ones, and how I check that the contracts are implementation-independent rather than just claiming they are. The problem the Floor is trying to help with Today, for most people, survival routes through financial access. To reach food, housing, energy, or healthcare you generally need money, and to hold or move money you need banking, employment, or purchasing power. Financial access has become the gate standing in front of the resources a person needs to stay alive. The goal of the Social Resource Floor is narrow and specific: to help make it so that financial status is not the condition that determines whether a person can reach the basic resources required to survive. It does not try to abolish money, banks, or markets — money stays a first-class resource and delivery method. It aims at one thing: a floor beneath which no person should fall, defined locally, reachable regardless of financial circumstances. That's the mission. Everything technical below exists to make that mission buildable by the institutions — governments, municipalities, NGOs, cooperatives, community providers — that would actually run it, without asking any of them to give up their own systems or hand over their data. Where the Floor sits The delivery systems for social protection already exist and are stron