标签：#distributed

A panel on data ownership challenged the definition of "ownership," arguing it must extend beyond simple account control to include structural independence, interoperability, and community governance. Speakers like Zenna Fiscella, Paul Frazee, Boris Mann, and Robin Berjon emphasised the need for shared standards, unbundled platforms, and better tools to support user sovereignty. By Olimpiu Pop

DoorDash details the architecture behind Ask DoorDash, its AI-powered conversational shopping assistant, combining LLMs, specialized AI agents, MCP-based tooling, and an intelligence layer with persistent consumer memory and live backend data. Early results show up to 24% higher checkout conversion, 17% larger baskets, and improved intent accuracy using memory-backed sessions. By Leela Kumili

Jake Lazaroff discussed the AT Protocol as a framework for distributed applications beyond social networking. He emphasised a local-first architecture where users maintain data in PDSs while leveraging shared infrastructure for synchronisation and updates. The presentation included experiments showcasing collaborative tools and highlighted the benefits of reduced reliance on app-specific backends. By Olimpiu Pop

Episode 1 established the mindset: failure is normal, not a sign of bad engineering. Episode 2 gets specific — you can't detect or handle a failure you can't even name. Saturday, Round 2 👦 Nephew: Uncle, last time you convinced me failure is basically guaranteed. Fine, I accept it. So what actually fails ? 👨‍🦳 Uncle: You tell me. Start listing things that could go wrong in your app right now. 👦 Nephew: Uh... the server could crash. The database could go down. My code could have a bug. 👨‍🦳 Uncle: Keep going. 👦 Nephew: The network? Someone could deploy the wrong thing? Payment gateway dies mid-checkout? 👨‍🦳 Uncle: You just named six of the seven categories without trying. You already know this. You've just never sorted it. 1. Hardware Failure 2. Software Failure 3. Network Failure 4. Database Failure 5. Third-Party Failure 6. Human Error 7. Resource Exhaustion 👦 Nephew: Then why do we need the list at all, if I already know it instinctively? 👨‍🦳 Uncle: Because "instinctively" isn't fast enough at 2 AM. Let's trace each one properly. Part 1 — Hardware Failure 👦 Nephew: This one's obvious anyway — I deploy to AWS. The cloud hides hardware failure from me. 👨‍🦳 Uncle: Does it? 👦 Nephew: ...doesn't it? That's the whole point of paying for EC2 instead of buying a server. 👨‍🦳 Uncle: Let's trace it. Your app sits on an EC2 instance. What's underneath the instance? 👦 Nephew: Virtual machine stuff, I guess? 👨‍🦳 Uncle: And underneath that ? 👦 Nephew: ...an actual physical machine somewhere. In a data center. 👨‍🦳 Uncle: There it is. Your app | "Virtual" server (EC2/Droplet) | ACTUAL physical hardware somewhere in a data center | Still capable of failing — just less visible to you 👦 Nephew: So it's not hidden. It's just one layer further away than I thought. 👨‍🦳 Uncle: Exactly. AWS absorbs a lot of it — that's part of what you're paying for — but disks still fail, instances still get abruptly terminated, whole availability zones still go down. That's Hardware Failure . Hardware Fa

Here are five architectural lessons we've learned from designing software for modern retailers.* Designing ERP Software for Retail: Five Lessons Every Software Engineer Should Know When people hear the word ERP , they often think of accounting software, dashboards, or inventory management. As software engineers, we see something different. We see distributed systems. Complex business workflows. Real-time data synchronization. Concurrent transactions. Event-driven architecture. And perhaps the biggest challenge of all—representing how real businesses actually operate. At RetailWings , we've learned that building an ERP for retail isn't simply a software engineering challenge. It's a business engineering challenge. Here are five lessons every engineer should understand before designing an ERP platform for modern retail. 1. Retail Doesn't Run in Modules—It Runs as One Business One of the biggest architectural mistakes in business software is treating departments as isolated applications. Many systems separate: Sales Inventory Finance Procurement HR But retailers don't experience their businesses that way. One sale immediately affects inventory. Inventory influences procurement. Procurement impacts finance. Finance drives reporting. Everything is connected. A well-designed ERP should reflect these relationships rather than forcing departments into disconnected silos. 2. Inventory Is More Than a Database Table To many engineers, inventory may appear to be a simple CRUD problem. Create. Read. Update. Delete. Retail quickly proves otherwise. Inventory changes through: Sales Returns Transfers Damages Procurement Stock adjustments Warehouse movements Manual reconciliations Every movement has financial implications. Every movement must be traceable. Designing inventory requires thinking in terms of events, not just records. 3. Real-Time Data Changes Everything Retail managers don't want yesterday's reports. They want answers now. How much stock is left? Which branch is sellin

If you've spent any time exploring backend engineering, distributed systems, or system design, you've almost certainly seen one book recommended more than any other: Designing Data-Intensive Applications , or DDIA for short. For years, I've heard experienced engineers describe it as the book that completely changed the way they think about software architecture. When the second edition was released with updated content covering modern distributed systems and cloud-native architectures, I decided it was finally time to see whether it deserved the hype. After reading it from beginning to end, I understand why this book has become a classic. It isn't another programming book that teaches a framework, a database, or a cloud platform. Instead, it teaches something much more valuable: how to think about building systems that continue working when data grows, traffic increases, and failures become inevitable. If you're a backend engineer—or want to become one—this is probably one of the best technical books you can read. This Isn't Really a Database Book The title can be a little misleading. Before opening DDIA, I assumed it would spend hundreds of pages comparing databases or discussing storage engines. Databases are certainly a major part of the discussion, but they're really just one piece of a much larger picture. The book is about designing systems that process enormous amounts of data while remaining reliable, scalable, and maintainable. Those systems happen to rely on databases, but they also involve replication, partitioning, distributed communication, stream processing, fault tolerance, consistency, messaging, and dozens of other architectural concepts that appear in modern software systems. By the end of the first few chapters, it becomes clear that the authors aren't trying to teach products. They're teaching engineering principles that remain useful no matter which technologies you're using. It Explains Why , Not Just How One of my favorite things about DDIA is

Instacart redesigned its personalized marketing system using a configuration-driven multi-tenant architecture on Storefront Pro. The system replaces retailer-specific implementations with a shared execution engine, enabling scalable personalization, faster configuration propagation in under a minute, and 99.9% delivery success across hundreds of retail banners through a unified campaign platform. By Leela Kumili

Modern Property Management Systems depend on continuous data exchange between internal modules and external services. Bookings, calendar updates, guest communication, cleaning tasks, and maintenance triggers all generate operational events that must be processed quickly and reliably. Free PMS platforms such as PMS.Rent rely on robust queueing and message‑broker layers to ensure that these events never get lost and are always processed in the correct order. At the core of this architecture is the concept of distributed message‑broker orchestration, which enables the PMS to scale horizontally, maintain predictable performance, and avoid bottlenecks during peak operational periods. Why Message Brokers Matter A PMS handles thousands of small but critical operations every day. Without a message broker, these operations would compete for system resources, causing delays, blocking workflows, and creating inconsistent states. A broker solves this by: receiving events, storing them durably, routing them to the correct processors, retrying failed operations, ensuring ordered execution when required. This creates a stable foundation for automation and real‑time synchronization. Queue Types Inside a PMS A modern PMS typically uses several queue types: Operational queues for bookings, calendar updates, and guest messages Automation queues for cleaning tasks, reminders, and workflow triggers Synchronization queues for channel managers and external APIs Fallback queues for events that require manual review Each queue isolates a specific category of tasks, preventing unrelated operations from interfering with each other. Distributed Workers Workers are lightweight processes that consume events from queues. They operate in parallel, allowing the PMS to scale dynamically. If the system detects increased load — for example, during high‑season booking spikes — it simply launches more workers. Workers typically perform tasks such as: updating property calendars, generating guest notific

Introduction In the world of distributed systems, complexity is the beast we’re all trying to tame. Teams building platforms often fall into the trap of believing that hiding this complexity is the ultimate goal. The logic seems sound: if users don’t see the mess, they won’t be burdened by it. But this approach, while well-intentioned, often leads to the creation of illusions —systems that appear simple on the surface but are brittle and unpredictable beneath. These illusions don’t just fail to solve the problem; they exacerbate it, leading to increased cognitive load, unexpected failures, and long-term maintenance nightmares. Consider a platform designed to abstract away the intricacies of distributed transactions. If the abstraction merely masks the complexity without addressing its root causes—such as inconsistent network latencies or partial failures—users will eventually encounter edge cases where the system behaves unpredictably. For example, a transaction might appear to succeed but fail silently due to a race condition in the underlying distributed lock mechanism. The illusion of simplicity breaks down when the system’s internal state deforms under pressure, leading to data inconsistencies or service outages. The core issue lies in the misunderstanding of abstractions . A meaningful abstraction doesn’t just hide complexity; it transforms it into a more manageable form. It exposes the essential properties of the system while encapsulating the non-essential details. In contrast, an illusion merely obscures the complexity, leaving it to fester beneath the surface. For instance, an abstraction might provide a consistent API for distributed state management, while internally handling retries, idempotency, and conflict resolution. An illusion, on the other hand, might simply wrap a flaky distributed database in a prettier interface, without addressing the underlying issues of consistency or availability. The pressure to deliver platforms quickly often exacerbates

Most engineers don't think seriously about Kafka partitioning until something breaks in production. A topic that worked fine at low volume starts falling behind. Events that should be in order aren't. All of it traces back to a partitioning decision that was made quickly and never revisited. Why Partitioning Actually Matters Partitions are the unit of parallelism in Kafka. Every consumer in a group is assigned one or more partitions, and it processes those partitions alone. No two consumers in the same group share a partition. That means your partition count sets a hard ceiling on how many consumers can work in parallel: if you have 6 partitions, the 7th consumer in your group sits idle no matter how much load you're under. Partitioning also controls ordering. Within a single partition, events are strictly ordered. Across partitions, there are no guarantees. So how you distribute events across partitions determines what ordering guarantees your consumers can actually rely on. Get this wrong and you'll spend a long time debugging why events from the same user are being processed out of sequence. The partition key controls both of these things. It determines which partition an event lands in, and that decision has consequences that are expensive to reverse. Partitioning Strategies Partition by Key This is the most common strategy and the right default when ordering matters. You supply a key when producing an event, Kafka hashes it using the murmur2 algorithm, and takes the modulo against the partition count to decide where it lands. producer . send ( ' orders ' , key = b ' user_4821 ' , value = event ) Every event with the same key always lands in the same partition. That's what guarantees ordering within a key. All events for user_4821 go to partition 3 (or wherever the hash resolves), and your consumer reads them in the exact sequence they were produced. I default to this for almost everything I build now and only go keyless when I have a specific reason to. Use key

A checkout saga spans inventory, payment, shipping, and loyalty. Downstream latency shifts every hour. Black Friday is not the day to discover your payment step timeout is baked into application.yml across twelve Spring Boot services. Kiponos.io gives every saga participant the same live orchestration parameters — step timeouts, retry budgets, compensation triggers — via one shared config tree. Each JVM reads locally on every saga step; ops adjusts once in the dashboard; WebSocket deltas propagate without redeploying the fleet. Why sagas break with static config Typical saga coordinator code: if ( step . elapsedMs () > 8000 ) { compensate ( "payment" , sagaId ); } That 8000 usually comes from: Per-service YAML — payment service says 8s, inventory says 12s; nobody agrees during an incident Env vars in Helm — change means rolling twelve deployments Shared DB config table — poll per step adds latency and coupling Saga steps are high-frequency reads inside workflow engines. You need local memory reads and async updates — the same contract as live API rate limits . Architecture: one tree, many participants ┌─────────────────┐ WebSocket deltas ┌──────────────────────┐ │ Kiponos.io UI │ ────────────────────────► │ Inventory service │ │ platform ops │ │ Payment service │ └─────────────────┘ │ Shipping service │ │ (each: in-mem SDK) │ └──────────┬───────────┘ │ .getInt() local ▼ ┌──────────────────────┐ │ saga step executor │ └──────────────────────┘ Every participant connects to profile ['orders']['v2']['prod']['sagas'] . When NOC extends payment.step_timeout_ms , all JVMs see the new value on the next step — no config server poll, no inter-service "what is timeout now?" REST calls. Shared saga config tree sagas/ checkout/ payment/ step_timeout_ms : 8000 max_retries : 2 retry_backoff_ms : 500 compensate_on_timeout : true inventory/ step_timeout_ms : 5000 max_retries : 3 hold_ttl_seconds : 120 shipping/ step_timeout_ms : 12000 fallback_carrier : ups_ground global/ saga_ttl_m

Slack has outlined how its AI serving infrastructure evolved through four distinct phases, moving from a self-managed Amazon SageMaker deployment to a multi-cloud architecture spanning AWS Bedrock and Google Cloud Vertex AI. By Matt Foster

Synchronous billing is easy, and that's the problem — it makes you think all billing is easy. When a request does its work inline, the billable number is in the response by the time you send it. The gateway meters from there — the meter write, retries and all, is its problem, not yours. From your side, synchronous billing is one number in the response. Asynchronous work breaks that. The request submits a job; the work happens later, in a worker; the result comes back through a poll or a callback. And the thing you bill for — characters processed, pages converted — isn't known when the request arrives. It's known when the job finishes . So you can't meter at the edge. The meter has to fire from the completion path. And the real difficulty is firing it exactly once per unit of completed work — because requests, polls, and retries all conspire to make that zero times or many times. This is platform-agnostic. Every submit-process-poll API has it. I'll use the system I run as the example, but the shape is the same anywhere. Three ways metering goes wrong On arrival. Carry the synchronous habit over and you meter when the job is submitted. But you don't know the size yet, so you're forced into a crude flat fee — or you bill for work that hasn't happened and might fail. Wrong unit, wrong time. On retrieval. The subtle one. You wire the meter to fire when the client fetches the result. Now a client who submits a job, lets it run — costing you real money downstream — and never bothers to poll is never billed. You did the work for free. "Completion" is not "the client picked up the result." It's the worker finishing. Without a fixed quantity. Input characters or output characters? Pages before OCR or after? If you haven't decided exactly what you measure and where, invoices drift and customers argue. Decide once; measure there. All three point the same way: meter on measured work-completion, with a fixed definition of the unit. Not on arrival. Not on retrieval. The mechanism:

Long-running workflows look simple when you first build them. Something happens. A few systems exchange data. Everything completes. Done. At least that's the expectation. Reality is very different. The biggest thing I underestimated was time. Not execution time. Elapsed time. Because once workflows start running for hours, days, or continuously, strange things start happening. APIs become temporarily unavailable Data changes halfway through the process Retries arrive much later than expected Someone manually updates a record Another system processes things in a different order Nothing is broken. But everything is slightly different from when the workflow started. Early on, I assumed workflows were transactions. Start. Execute. Finish. Now I think of them as conversations between systems. And conversations can get interrupted. Another thing I underestimated: State changes. You might start processing an order that is "pending". Ten minutes later, another system marks it as "cancelled". An hour later, a retry comes in from an earlier step. If your workflow only thinks about data, weird things happen. Because the world has changed while the process was still running. Long-running workflows also expose assumptions you didn't know you made. Like: this API will always respond quickly data will arrive in order users won't modify records manually retries will happen immediately Those assumptions survive in testing. Production removes them quickly. One thing that changed how I build these systems: I stopped asking: "Will this workflow finish?" And started asking: "What state will the world be in when it finishes?" Because those are two very different questions. Most problems in long-running systems aren't caused by one big failure. They're caused by lots of small changes happening while the workflow is still alive. And if you don't account for that, eventually the workflow finishes successfully and still produces the wrong outcome. This is something we think about constantly

Introduction This is the second part of sharing my journey to create a microservices-based backend system, in the previous part, I introduced the system and its services. In this part I am going to mention the main things that happened with me from then until now. Transitioning to online DBaaS platforms The first main thing was to transition my databases from my local device to accessible over the internet. I used Neon and MongoDB Atlas . Although performance-wise it is better to deploy the database on the same server that the whole system so all services can access the databases without network overhead , This solution is better from a point, that is avoiding consuming the free VM instance that I got to host my main services, as with DBaaS platforms I will not require storage for databases or additional overhead to handle the DBMS on a free small resources-constrained VM. Observability Observability is one of the main concerns in any system, it gives the ability for the system to expose what happens inside it without any need to guess and manually trace the code to determine where the error may happened. I utilized OpenTelemetry , as I found it is most used and accepted tool to log, trace and collect metrics about the system and the traffic. Also, I loved that it is not related to specific framework, that is .NET by the way, I loved the idea that it is standalone tool. But logging and tracing and metrics collection will not be beneficial if we do not see it actually and can achieve monitoring from those telemetry data, so I used Grafana Cloud to export the telemetry data to some place that is accessible online and a tool that can generate dashboards and visualizing for the metrics ( Prometheus ), and UI that can show all the telemetry data easily. Deployment Until this point, we have a system that is fully functional and have consistent docker configuration for its services via Docker Compose, and it database is already there and available. Now this is the point at

The Problem: Feature Velocity Was Creating Structural Debt The system originally started as a simple feature delivery backend: A Django API powering agricultural insights Celery workers handling asynchronous processing Independent endpoints for each new capability A growing set of Earth Observation computations (NDVI, NDWI, etc.) At first, it worked. But as more features were added, a pattern emerged: Each feature introduced its own pipeline logic Observability was inconsistent across services API contracts drifted between frontend and backend Debugging required tracing multiple disconnected systems We weren’t scaling functionality. We were scaling fragmentation. The Turning Point: Features vs Platforms The key realization was simple: Features solve user problems. Platforms solve system problems. We were repeatedly rebuilding: Authentication flows Data ingestion logic Processing pipelines API validation layers Monitoring hooks Each feature was solving its own version of these concerns. That is where platform engineering became necessary. The Shift: Introducing a Platform Layer We introduced a platform layer between feature delivery and infrastructure. Instead of building isolated pipelines, we standardized: 1. Unified API Surface All Earth Observation workflows (NDVI, NDWI, and future indices) were normalized into a consistent API contract. Shared request/response structure Versioned endpoints Schema validation through serializers Central routing logic This eliminated endpoint fragmentation. 2. Standardized Processing Pipeline Celery tasks were refactored into a reusable pipeline pattern: Ingestion Validation Computation Storage Publishing Instead of feature-specific workers, we moved toward composable tasks. This allowed new indices or processing logic to plug into the same execution flow. 3. Observability as a First-Class Layer One of the biggest failures in the original system was visibility. We introduced: Structured logging across all services Traceable job IDs

I bombed a system design interview last year — not because I didn't know the architecture, but because I spent the first 5 minutes fighting Excalidraw. So I built SystemDesignBoard — a free, keyboard-first whiteboard specifically for system design interviews. What it does You open it, press a key, and start drawing. No account, no onboarding, no drag-from-a-sidebar friction. R → place a Service node C → place a Database/Cache/Queue A → connect two nodes N → open the scratchpad for scale math The features I'm most proud of Animated connectors that show communication type Instead of just drawing arrows, connectors visually encode how services talk: ⇄ sync — paired dashes (request + ACK) ≋ stream — near-solid fast line with glow (continuous pipeline) This matters in interviews — your interviewer can glance at your diagram and immediately understand the communication pattern. Cloud provider badges Tag any node as AWS (EC2, Lambda, RDS, S3), GCP (GKE, Cloud Run, Firestore), or Azure. Each subtype has its own icon. Trade-off logging Right-click any node → Log Trade-offs → attach your CAP theorem stance, consistency level, and scaling strategy directly to the component. Diagram-as-Code Type: [Mobile App] -> [API Gateway] [API Gateway] -> [Auth Service] [Auth Service] -> [Users DB] [Feed Service] -> [Posts DB x3] [Feed Service] -> [Redis Cache] Hit Apply — it auto-lays out the whole architecture in seconds. Export to animated GIF Export your diagram as a GIF that shows live traffic flow animations. Great for sharing after an interview or in a design doc. Tech stack React + TypeScript + Vite @xyflow/react (ReactFlow v12) for the canvas Zustand + Immer for state with full undo/redo html-to-image + gifshot for PNG/GIF export It's free and open No signup required. Works entirely in the browser. Free during beta. 👉 systemdesignboard.com Would love feedback — especially from anyone who's done system design interviews recently. What's missing? What's annoying? Drop a comment below

The Counterintuitive Truth: Feature Flags Are Not Config Files Most engineers first encounter feature flags as a simple abstraction: a key-value lookup that returns true or false. That mental model works fine for a single service handling a few hundred requests per minute. It becomes actively dangerous at scale. A mature feature flag system isn't a config file with an API wrapper — it's a distributed control plane . The distinction matters architecturally. A control plane manages the real-time behavior of a running system across many nodes simultaneously, with its own consistency guarantees, failure semantics, and propagation latency. That's a fundamentally different design problem than reading a YAML file on startup. One constraint drives every downstream decision: user traffic must never block on a remote flag service call. If evaluation requires a synchronous RPC, you've coupled your request path to the availability and latency of an external system. Netflix's Archaius library enforces this by evaluating flags entirely in-process against a locally-cached configuration snapshot. A network round-trip per evaluation injects 10–50ms of tail latency at p99 — catastrophic when you're competing on streaming start times measured in hundreds of milliseconds. Google, Meta, and Netflix collectively evaluate flags against millions of requests per second with sub-millisecond overhead. That figure is only achievable through local evaluation backed by an async synchronization layer, not RPC. The other failure mode engineers underestimate is flag sprawl . Systems accumulate flags the way codebases accumulate dead functions — gradually, then all at once. I've seen services carrying thousands of flags where fewer than 10% were actively managed. The operational weight alone becomes a liability: which flags are safe to remove? Which ones are kill switches for production behavior that no one documented? Knight Capital's $440M loss in 45 minutes in 2012 remains the canonical cautionar

Failures are obvious. Retries are sneaky. When something fails, everyone notices. An alert goes off. A request errors out. Someone starts investigating. Retries are different. They look harmless. Most of the time, they save the system. But sometimes, retries create bigger problems than the original failure. Imagine an API call times out. No problem. The system retries. But what if the first request actually succeeded and only the response was lost? Now the retry creates: duplicate orders repeated emails inconsistent records workflows running twice The failure happened once. The retry multiplied it. Another thing I've seen: One slow dependency causes requests to pile up. Retries start firing. Those retries create even more traffic. Which slows things down further. Which triggers even more retries. Suddenly, the system is spending more effort retrying than doing useful work. Retries also hide problems. A temporary issue gets retried five times and eventually succeeds. Everything looks normal. Meanwhile: latency increases queues grow users experience delays Nothing technically failed. But the system is getting less healthy. What changed for me is that I stopped treating retries as free. Every retry has a cost. It consumes resources. It increases load. And if actions aren't designed carefully, retries can repeat side effects that should only happen once. Now when I build something, I don't ask: "What happens if this fails?" I ask: "What happens if this runs again?" Because in production, things almost always run again. And if the answer is "bad things happen," the retry mechanism isn't helping. It's making things worse. Failures are part of every system. Retries are too. The difference is that failures usually happen once. Retries can turn one problem into hundreds if you don't design for them. This is something we think about constantly at BrainPack when operating long-running workflows across multiple systems. AI and automation layers make retries even more common, wh

pgrac is an open attempt to rebuild Oracle RAC's core machinery (shared-everything storage, multiple active nodes all writing one database, a cluster-wide change number) on top of PostgreSQL 16. Build log #1 laid out the four problems that fight back. This one is about the problem that turns a node failure into silent data corruption, and the first, deliberately modest, layer pgrac ships against it. The failure mode In a shared-nothing cluster an evicted node is mostly harmless: it owns its own disks, so the cluster routes around it. In a shared-everything cluster the same event is dangerous, because every node writes the same storage. Picture the classic split: node 2 misses heartbeats, the cluster declares it dead and remasters its work elsewhere, but node 2 is not actually dead. It is frozen on a long GC pause, or its interconnect NIC flaked, and it is about to wake up and finish the write it started. Now two nodes believe they own the same blocks, and shared storage will accept both writes. That is not a crash. It is corruption you find three days later. Oracle RAC's answer is I/O fencing: before remastering a dead node's resources, you make certain it can no longer touch the storage, with STONITH, SCSI-3 persistent reservations, or a hardware watchdog. The node is fenced at a layer below its own software, because the whole point is that you cannot trust the dead node's software to behave. That hardware layer is real work, and it is not what pgrac built first. What it built first is the layer above it: an in-process cooperative write-fence, now default-ON. The rest of this is precise about what that does and does not buy you, because "we have fencing" is the kind of claim that is worth less than nothing if it is overstated. A fence needs an authority everyone can agree on You cannot fence on local opinion, because the whole problem is that the dead node disagrees about being dead. Authority has to live on durable, shared, quorum-backed storage. pgrac writes a sm