标签：#distributedsystems

Originally published on nejckorasa.github.io . When a team moves from a monolith into microservices and event-driven, asynchronous systems, it inherits a class of problems that used to be someone else's: work that fails halfway through, steps that must not run twice, calls that return before the work is done. Temporal is a durable execution engine that handles a lot of this - you define a multi-step process, and it guarantees the process runs to completion even when workers crash in the middle. I've spent the better part of a decade building distributed systems in the money-movement core of banks - ledgers, payments, credit cards - a lot of it on Temporal, from short request-triggered workflows to ones that stayed open for weeks. This is the high-level guide I'd give a team making that jump: the principles worth internalising before you ship, not a full tutorial. Most of them aren't really about Temporal. They're the habits the async shift demands - Temporal just punishes you quickly when you skip one. Durable Execution: The Problem It Solves Distributed work fails in the middle. You call service A, it succeeds. You call B, it times out. The pod dies before C. Now you have half-finished work and no memory of how far you got. The usual fix is a pile of status columns, a cron job to find stuck rows, and retry logic hand-rolled for every step. Temporal's promise is that any process you start runs to the end. The runtime picture: there's a Temporal service (its own cluster), and your app runs worker processes that poll it and execute your code. As a workflow runs, Temporal records every step to an event history . If a worker dies, another picks the workflow up and replays that history to rebuild state, then carries on from where it left off, retrying anything that failed. The history is the source of truth, and it survives the crash. Most of the rules below fall out of that one fact. The Golden Rule: Workflows Decide, Activities Do There are two kinds of code in Tempora

This is going to be a very different article from what I usually write. No technical discussions, architecture deep dives, or engineering practices today. Instead, we're talking about something much older than software itself: music. We treat language like it's the default mode of human communication, like it's the real and only thing used to communicate, everything else is secondary, emotional, aesthetic, nice to have. But language is actually the outlier. It's the new protocol layered on top of something much older. Music is the original standard and we've basically forgotten how to read it. The Protocol Stack Think of communication like a network stack. Language is high-level. It's TCP/IP. Built on assumptions, needs learning, breaks the second you cross a boundary. You need: A shared vocabulary Syntactic understanding Cultural context Years of study if you actually want fluency It's powerful but It's also fragile. And it's recent . Written language is a few thousand years old. Spoken language is older, sure, but both are late abstractions compared to the hundreds of thousands of years humans have been syncing bodies to shared sound. Relative to that timeline? Language is yesterday's patch. Music? That's the lower-level protocol. The physical layer everything else runs on. A Japanese teenager at a Michael Jackson concert doesn't need to speak English. She doesn't need to understand what "Man in the Mirror" means as a concept. She also doesn't need a music degree. Music isn't zero -cost. Genre, culture, convention still shape how we hear it. But the entry barrier for emotional communication is way lower. A rhythm can hit urgency, celebration, sadness, or tension long before anyone understands the formal structure behind it. Her nervous system speaks that fluently. And so does everyone else in that stadium. How the Protocol Works Here's what happens when the song starts: 70,000 people stop being individuals and start being a distributed system synchronizing to the

Controller A records revocation epoch 12. Worker B, partitioned with a cached grant from epoch 11, starts another external action. The database is correct and the system is unsafe. Emergency stop is therefore a distributed protocol, not a Boolean field. What is verified In its July 21 disclosure, OpenAI says an internal benchmark used models with reduced cyber refusals and that a combination of models compromised Hugging Face infrastructure. The primary source is https://openai.com/index/hugging-face-model-evaluation-security-incident/ . Reporting on July 24 then described US discussion of emergency-shutdown and independent-audit proposals. The latter is policy coverage, not enacted law and not an extension of the official incident facts. Missing protocol details, impact boundaries, and remediation should remain unknown rather than inferred. Invariants and assumptions Assume workers, queue consumers, an authorization service, and external adapters can fail independently. Messages may be delayed, duplicated, or reordered; clocks have bounded error only if measured. Required invariants: No action starts with a grant epoch below the subject's revocation epoch. Cached grants expire within a declared lease bound. Restart cannot lower a persisted epoch. Duplicate revocation converges to the same or higher epoch. Completion means every registered executor acknowledged or its lease expired. revoke(subject, epoch=13) -> durable CAS max(current, 13) -> publish {subject, epoch:13} -> executors persist max(local, 13), ack -> controller waits for ack set OR lease expiry -> issue completion receipt with missing/expired members Failure injection Property Acceptance rule delay revocation event lease bounds stale authority no start after local lease expiry duplicate epoch 13 idempotence epoch remains 13+ deliver 13 before 12 monotonicity never returns to 12 worker restarts durability loads persisted epoch before work controller partition fail closed no new lease after expiry A minim

When a new video shows up in one of our regional crawlers, three things need to happen almost immediately: the SQLite FTS5 search index for that region needs a new row, the discovery ranking cache needs to be invalidated, and the sitemap generator needs to know a URL was born. For a long time we did all of this inline, inside the same cron process that fetched the video. It worked until it didn't. A slow FTS5 rebuild would stall the fetch loop, a sitemap write would fail silently, and a crash halfway through meant one region had the video indexed and another didn't. The fetch and the fan-out were fused together, and every failure was a partial failure. The fix was to stop treating "a video was discovered" as a function call and start treating it as an event. At TrendVidStream we run discovery across 8 regions, and the moment we introduced NATS JetStream as the spine between the crawler and the downstream consumers, the whole system got calmer. This post is the concrete version of how we did it: the stream config, the publishers, the consumers, and the mistakes we made that you can skip. Why Not Just Use a Queue Table in SQLite We already had SQLite everywhere, so the obvious move was a jobs table. We tried it. The problems showed up fast: Polling latency vs. load tradeoff. Poll every second and you hammer the DB with mostly-empty SELECT queries across 8 regions. Poll every 30 seconds and your search index lags noticeably behind your crawler. No fan-out. One row, one worker. If the sitemap generator and the FTS5 indexer both need the same event, you either duplicate rows or invent a consumed_by bitmask. Both are ugly. Locking. SQLite's writer lock means the queue table and the actual data table start contending under the multi-region cron bursts we run. Cross-region delivery. Our regions aren't all on the same box. A queue table doesn't cross machines without you building a replication story on top. JetStream solves all four: push-based delivery (no polling), multipl

In this article, we're going to explore Event Schema evolution with Event versioning 10. Event Schemas Will Eventually Change No event schema stays the same forever. As businesses grow, regulations shift, products gain new features, and processes become more complex, the data shared between services must evolve as well. This evolution is not optional—it is a natural consequence of a system adapting to changing requirements. Many teams initially assume they can simply update an event whenever needed. This assumption may hold when there is only one producer and one consumer, but real-world systems rarely remain that simple. Over time, multiple consumers emerge, each with its own responsibilities and release cycles. A typical system often looks like this: OrderConfirmed | +------------------+-------------------+ | | | v v v Inventory Billing Notification | v Analytics | v Customer Insights Each consumer evolves independently. Some services may deploy updates weekly, while others might release changes quarterly. In some cases, consumers may even belong to external teams with entirely different priorities and timelines. Because of this, producers cannot assume that all consumers will upgrade simultaneously. Schema evolution, therefore, is not just about modifying data structures. It is fundamentally about maintaining compatibility across independently evolving systems. Compatibility Is More Important Than Version Numbers When discussing schema evolution, teams often focus immediately on versioning. While versioning is useful, compatibility is far more critical. Without compatibility, versioning alone cannot prevent system breakage. Consider the following event: { "orderId" : "ORD-1001" , "customerId" : "CUS-501" , "totalAmount" : 249.99 } Now imagine a new requirement introduces currency. One approach might replace the existing field entirely: { "orderId" : "ORD-1001" , "customerId" : "CUS-501" , "amount" : { "value" : 249.99 , "currency" : "USD" } } Although the data mo

I want to start with a moment most of us have lived through. It's 3 a.m. A dashboard is red. You're eight terminals deep in grep , trying to work out which service actually fell over and why. And the whole time there's this nagging feeling that you're doing archaeology on a system you wrote last month. Here's what got under my skin about it. The answer was never actually lost. Back in the source code it said, in plain terms, that the payment service talks to Postgres through a connection pool. That this retry backs off three times. That this particular dependency is external and you must never, ever try to "just restart it." That was all right there at build time. Then we packaged everything up, deployed, threw that structure in the bin, and asked a sleep-deprived human to reconstruct it from log lines. That gap bugged me enough that I spent a while building something around it. This post is about that. Autoscaling is good at the wrong problem We've gotten genuinely good at reacting to resource pressure. Traffic climbs, a box gets slow, CPU pins, and the autoscaler adds capacity or sheds load. No complaints there, it's kept things running for years. The problem is it has no idea what your app is for . It can't tell a service that's slow because it's healthy and hammered from a service that's fast because it's quietly writing garbage to the database. It never had a model of the application in the first place. So a whole category of failures just sails right past it. A connection pool getting drained by something downstream. A poison message kicking off a retry storm. A schema change that breaks one code path and leaves the other one looking perfectly fine. Infrastructure that only thinks in CPU and memory is blind to all of that. The "throw an LLM at it" era The going answer right now is to bolt a large language model onto your observability stack. Fire hose all the logs, traces, and metrics at a big central model and ask it what happened. I get why. I also think it'

"We're using Kafka with exactly-once semantics, so we don't have to worry about duplicates." I've heard this in architecture reviews, design docs, and postmortem explanations. It represents a misunderstanding of what Kafka's exactly-once guarantee actually covers, and the gap between the promise and the reality has caused real production incidents. What Kafka's Exactly-Once Actually Covers Kafka's exactly-once semantics (EOS), introduced in 0.11.0, operates at two levels: Producer idempotence ( enable.idempotence=true ): The producer assigns each message a sequence number. The broker deduplicates messages with the same producer ID and sequence number. This prevents duplicates caused by producer retries — the message lands in the Kafka partition exactly once, regardless of retry count. Transactions ( transactional.id ): Allows a producer to write to multiple partitions atomically. Either all writes commit or none do. Combined with isolation.level=read_committed on consumers, readers only see committed transactions. Together, these give you exactly-once message delivery within the Kafka cluster. What Exactly-Once Does Not Cover Here's the boundary that engineers miss: Kafka's exactly-once guarantee is scoped to the Kafka cluster. The moment your consumer does anything outside Kafka — writes to a database, calls a REST API, publishes to a cloud queue — you're outside the transaction boundary. Consider a typical consumer: consumer . poll ( records ); for ( record : records ) { database . save ( process ( record )); // External write — outside Kafka transaction } consumer . commitSync (); If the application crashes after database.save() but before commitSync() , Kafka re-delivers the message. The consumer reprocesses it. The database now has two writes for the same event. Enabling producer idempotence on the consumer's Kafka writes does not fix this. The Patterns That Actually Give You End-to-End Safety Idempotent Consumers Design consumer processing logic to be idempote

Changing the model for a running AI task is not a settings update. It is a distributed operation: read current task -> prepare credentials/config -> request restart -> receive result -> persist active model If two switches overlap, completion order can differ from request order. The system needs a rule for which intent wins. The concrete case At commit c58bcd4 , MonkeyCode records model-switch attempts with from/to model IDs, request ID, load-session flag, success, message, session ID, and timestamps in TaskModelSwitch . The reviewed task use case creates a switch record, asks taskflow to restart with the target model configuration, and completes the switch record and task model based on the response. The accompanying tests cover success and failure paths. From this source review, I could not establish an explicit compare-and-swap generation or a per-task serialization contract around overlapping requests. That does not prove an exploitable race: serialization may exist elsewhere in the deployment or taskflow boundary. It means concurrency semantics deserve an explicit test and contract. Why last completion is unstable Assume request A selects model A, then request B selects model B: time -> A: request ---- restart ---------------- complete B: request -- restart -- complete If each successful completion writes its model, B applies first and late A overwrites it. Reverse network timing and the result changes. The companion simulator makes that order dependence visible: export function naiveCompletionOrder ( completions ) { let model = " initial " ; for ( const completion of completions ) { if ( completion . success ) model = completion . model ; } return model ; } [A, B] ends on B. [B, A] ends on A. The caller's latest intent is not part of the rule. Add a monotonic generation Assign a generation while accepting each request: A -> generation 41 B -> generation 42 Completion may update active state only when its generation equals the task's current requested generatio

Episode 1 established the mindset: failure is normal, not a sign of bad engineering. Episode 2 gets specific — you can't detect or handle a failure you can't even name. Saturday, Round 2 👦 Nephew: Uncle, last time you convinced me failure is basically guaranteed. Fine, I accept it. So what actually fails ? 👨‍🦳 Uncle: You tell me. Start listing things that could go wrong in your app right now. 👦 Nephew: Uh... the server could crash. The database could go down. My code could have a bug. 👨‍🦳 Uncle: Keep going. 👦 Nephew: The network? Someone could deploy the wrong thing? Payment gateway dies mid-checkout? 👨‍🦳 Uncle: You just named six of the seven categories without trying. You already know this. You've just never sorted it. 1. Hardware Failure 2. Software Failure 3. Network Failure 4. Database Failure 5. Third-Party Failure 6. Human Error 7. Resource Exhaustion 👦 Nephew: Then why do we need the list at all, if I already know it instinctively? 👨‍🦳 Uncle: Because "instinctively" isn't fast enough at 2 AM. Let's trace each one properly. Part 1 — Hardware Failure 👦 Nephew: This one's obvious anyway — I deploy to AWS. The cloud hides hardware failure from me. 👨‍🦳 Uncle: Does it? 👦 Nephew: ...doesn't it? That's the whole point of paying for EC2 instead of buying a server. 👨‍🦳 Uncle: Let's trace it. Your app sits on an EC2 instance. What's underneath the instance? 👦 Nephew: Virtual machine stuff, I guess? 👨‍🦳 Uncle: And underneath that ? 👦 Nephew: ...an actual physical machine somewhere. In a data center. 👨‍🦳 Uncle: There it is. Your app | "Virtual" server (EC2/Droplet) | ACTUAL physical hardware somewhere in a data center | Still capable of failing — just less visible to you 👦 Nephew: So it's not hidden. It's just one layer further away than I thought. 👨‍🦳 Uncle: Exactly. AWS absorbs a lot of it — that's part of what you're paying for — but disks still fail, instances still get abruptly terminated, whole availability zones still go down. That's Hardware Failure . Hardware Fa

Here are five architectural lessons we've learned from designing software for modern retailers.* Designing ERP Software for Retail: Five Lessons Every Software Engineer Should Know When people hear the word ERP , they often think of accounting software, dashboards, or inventory management. As software engineers, we see something different. We see distributed systems. Complex business workflows. Real-time data synchronization. Concurrent transactions. Event-driven architecture. And perhaps the biggest challenge of all—representing how real businesses actually operate. At RetailWings , we've learned that building an ERP for retail isn't simply a software engineering challenge. It's a business engineering challenge. Here are five lessons every engineer should understand before designing an ERP platform for modern retail. 1. Retail Doesn't Run in Modules—It Runs as One Business One of the biggest architectural mistakes in business software is treating departments as isolated applications. Many systems separate: Sales Inventory Finance Procurement HR But retailers don't experience their businesses that way. One sale immediately affects inventory. Inventory influences procurement. Procurement impacts finance. Finance drives reporting. Everything is connected. A well-designed ERP should reflect these relationships rather than forcing departments into disconnected silos. 2. Inventory Is More Than a Database Table To many engineers, inventory may appear to be a simple CRUD problem. Create. Read. Update. Delete. Retail quickly proves otherwise. Inventory changes through: Sales Returns Transfers Damages Procurement Stock adjustments Warehouse movements Manual reconciliations Every movement has financial implications. Every movement must be traceable. Designing inventory requires thinking in terms of events, not just records. 3. Real-Time Data Changes Everything Retail managers don't want yesterday's reports. They want answers now. How much stock is left? Which branch is sellin

If you've spent any time exploring backend engineering, distributed systems, or system design, you've almost certainly seen one book recommended more than any other: Designing Data-Intensive Applications , or DDIA for short. For years, I've heard experienced engineers describe it as the book that completely changed the way they think about software architecture. When the second edition was released with updated content covering modern distributed systems and cloud-native architectures, I decided it was finally time to see whether it deserved the hype. After reading it from beginning to end, I understand why this book has become a classic. It isn't another programming book that teaches a framework, a database, or a cloud platform. Instead, it teaches something much more valuable: how to think about building systems that continue working when data grows, traffic increases, and failures become inevitable. If you're a backend engineer—or want to become one—this is probably one of the best technical books you can read. This Isn't Really a Database Book The title can be a little misleading. Before opening DDIA, I assumed it would spend hundreds of pages comparing databases or discussing storage engines. Databases are certainly a major part of the discussion, but they're really just one piece of a much larger picture. The book is about designing systems that process enormous amounts of data while remaining reliable, scalable, and maintainable. Those systems happen to rely on databases, but they also involve replication, partitioning, distributed communication, stream processing, fault tolerance, consistency, messaging, and dozens of other architectural concepts that appear in modern software systems. By the end of the first few chapters, it becomes clear that the authors aren't trying to teach products. They're teaching engineering principles that remain useful no matter which technologies you're using. It Explains Why , Not Just How One of my favorite things about DDIA is

Modern Property Management Systems depend on continuous data exchange between internal modules and external services. Bookings, calendar updates, guest communication, cleaning tasks, and maintenance triggers all generate operational events that must be processed quickly and reliably. Free PMS platforms such as PMS.Rent rely on robust queueing and message‑broker layers to ensure that these events never get lost and are always processed in the correct order. At the core of this architecture is the concept of distributed message‑broker orchestration, which enables the PMS to scale horizontally, maintain predictable performance, and avoid bottlenecks during peak operational periods. Why Message Brokers Matter A PMS handles thousands of small but critical operations every day. Without a message broker, these operations would compete for system resources, causing delays, blocking workflows, and creating inconsistent states. A broker solves this by: receiving events, storing them durably, routing them to the correct processors, retrying failed operations, ensuring ordered execution when required. This creates a stable foundation for automation and real‑time synchronization. Queue Types Inside a PMS A modern PMS typically uses several queue types: Operational queues for bookings, calendar updates, and guest messages Automation queues for cleaning tasks, reminders, and workflow triggers Synchronization queues for channel managers and external APIs Fallback queues for events that require manual review Each queue isolates a specific category of tasks, preventing unrelated operations from interfering with each other. Distributed Workers Workers are lightweight processes that consume events from queues. They operate in parallel, allowing the PMS to scale dynamically. If the system detects increased load — for example, during high‑season booking spikes — it simply launches more workers. Workers typically perform tasks such as: updating property calendars, generating guest notific

Introduction In the world of distributed systems, complexity is the beast we’re all trying to tame. Teams building platforms often fall into the trap of believing that hiding this complexity is the ultimate goal. The logic seems sound: if users don’t see the mess, they won’t be burdened by it. But this approach, while well-intentioned, often leads to the creation of illusions —systems that appear simple on the surface but are brittle and unpredictable beneath. These illusions don’t just fail to solve the problem; they exacerbate it, leading to increased cognitive load, unexpected failures, and long-term maintenance nightmares. Consider a platform designed to abstract away the intricacies of distributed transactions. If the abstraction merely masks the complexity without addressing its root causes—such as inconsistent network latencies or partial failures—users will eventually encounter edge cases where the system behaves unpredictably. For example, a transaction might appear to succeed but fail silently due to a race condition in the underlying distributed lock mechanism. The illusion of simplicity breaks down when the system’s internal state deforms under pressure, leading to data inconsistencies or service outages. The core issue lies in the misunderstanding of abstractions . A meaningful abstraction doesn’t just hide complexity; it transforms it into a more manageable form. It exposes the essential properties of the system while encapsulating the non-essential details. In contrast, an illusion merely obscures the complexity, leaving it to fester beneath the surface. For instance, an abstraction might provide a consistent API for distributed state management, while internally handling retries, idempotency, and conflict resolution. An illusion, on the other hand, might simply wrap a flaky distributed database in a prettier interface, without addressing the underlying issues of consistency or availability. The pressure to deliver platforms quickly often exacerbates

Most engineers don't think seriously about Kafka partitioning until something breaks in production. A topic that worked fine at low volume starts falling behind. Events that should be in order aren't. All of it traces back to a partitioning decision that was made quickly and never revisited. Why Partitioning Actually Matters Partitions are the unit of parallelism in Kafka. Every consumer in a group is assigned one or more partitions, and it processes those partitions alone. No two consumers in the same group share a partition. That means your partition count sets a hard ceiling on how many consumers can work in parallel: if you have 6 partitions, the 7th consumer in your group sits idle no matter how much load you're under. Partitioning also controls ordering. Within a single partition, events are strictly ordered. Across partitions, there are no guarantees. So how you distribute events across partitions determines what ordering guarantees your consumers can actually rely on. Get this wrong and you'll spend a long time debugging why events from the same user are being processed out of sequence. The partition key controls both of these things. It determines which partition an event lands in, and that decision has consequences that are expensive to reverse. Partitioning Strategies Partition by Key This is the most common strategy and the right default when ordering matters. You supply a key when producing an event, Kafka hashes it using the murmur2 algorithm, and takes the modulo against the partition count to decide where it lands. producer . send ( ' orders ' , key = b ' user_4821 ' , value = event ) Every event with the same key always lands in the same partition. That's what guarantees ordering within a key. All events for user_4821 go to partition 3 (or wherever the hash resolves), and your consumer reads them in the exact sequence they were produced. I default to this for almost everything I build now and only go keyless when I have a specific reason to. Use key

A checkout saga spans inventory, payment, shipping, and loyalty. Downstream latency shifts every hour. Black Friday is not the day to discover your payment step timeout is baked into application.yml across twelve Spring Boot services. Kiponos.io gives every saga participant the same live orchestration parameters — step timeouts, retry budgets, compensation triggers — via one shared config tree. Each JVM reads locally on every saga step; ops adjusts once in the dashboard; WebSocket deltas propagate without redeploying the fleet. Why sagas break with static config Typical saga coordinator code: if ( step . elapsedMs () > 8000 ) { compensate ( "payment" , sagaId ); } That 8000 usually comes from: Per-service YAML — payment service says 8s, inventory says 12s; nobody agrees during an incident Env vars in Helm — change means rolling twelve deployments Shared DB config table — poll per step adds latency and coupling Saga steps are high-frequency reads inside workflow engines. You need local memory reads and async updates — the same contract as live API rate limits . Architecture: one tree, many participants ┌─────────────────┐ WebSocket deltas ┌──────────────────────┐ │ Kiponos.io UI │ ────────────────────────► │ Inventory service │ │ platform ops │ │ Payment service │ └─────────────────┘ │ Shipping service │ │ (each: in-mem SDK) │ └──────────┬───────────┘ │ .getInt() local ▼ ┌──────────────────────┐ │ saga step executor │ └──────────────────────┘ Every participant connects to profile ['orders']['v2']['prod']['sagas'] . When NOC extends payment.step_timeout_ms , all JVMs see the new value on the next step — no config server poll, no inter-service "what is timeout now?" REST calls. Shared saga config tree sagas/ checkout/ payment/ step_timeout_ms : 8000 max_retries : 2 retry_backoff_ms : 500 compensate_on_timeout : true inventory/ step_timeout_ms : 5000 max_retries : 3 hold_ttl_seconds : 120 shipping/ step_timeout_ms : 12000 fallback_carrier : ups_ground global/ saga_ttl_m

Synchronous billing is easy, and that's the problem — it makes you think all billing is easy. When a request does its work inline, the billable number is in the response by the time you send it. The gateway meters from there — the meter write, retries and all, is its problem, not yours. From your side, synchronous billing is one number in the response. Asynchronous work breaks that. The request submits a job; the work happens later, in a worker; the result comes back through a poll or a callback. And the thing you bill for — characters processed, pages converted — isn't known when the request arrives. It's known when the job finishes . So you can't meter at the edge. The meter has to fire from the completion path. And the real difficulty is firing it exactly once per unit of completed work — because requests, polls, and retries all conspire to make that zero times or many times. This is platform-agnostic. Every submit-process-poll API has it. I'll use the system I run as the example, but the shape is the same anywhere. Three ways metering goes wrong On arrival. Carry the synchronous habit over and you meter when the job is submitted. But you don't know the size yet, so you're forced into a crude flat fee — or you bill for work that hasn't happened and might fail. Wrong unit, wrong time. On retrieval. The subtle one. You wire the meter to fire when the client fetches the result. Now a client who submits a job, lets it run — costing you real money downstream — and never bothers to poll is never billed. You did the work for free. "Completion" is not "the client picked up the result." It's the worker finishing. Without a fixed quantity. Input characters or output characters? Pages before OCR or after? If you haven't decided exactly what you measure and where, invoices drift and customers argue. Decide once; measure there. All three point the same way: meter on measured work-completion, with a fixed definition of the unit. Not on arrival. Not on retrieval. The mechanism:

Long-running workflows look simple when you first build them. Something happens. A few systems exchange data. Everything completes. Done. At least that's the expectation. Reality is very different. The biggest thing I underestimated was time. Not execution time. Elapsed time. Because once workflows start running for hours, days, or continuously, strange things start happening. APIs become temporarily unavailable Data changes halfway through the process Retries arrive much later than expected Someone manually updates a record Another system processes things in a different order Nothing is broken. But everything is slightly different from when the workflow started. Early on, I assumed workflows were transactions. Start. Execute. Finish. Now I think of them as conversations between systems. And conversations can get interrupted. Another thing I underestimated: State changes. You might start processing an order that is "pending". Ten minutes later, another system marks it as "cancelled". An hour later, a retry comes in from an earlier step. If your workflow only thinks about data, weird things happen. Because the world has changed while the process was still running. Long-running workflows also expose assumptions you didn't know you made. Like: this API will always respond quickly data will arrive in order users won't modify records manually retries will happen immediately Those assumptions survive in testing. Production removes them quickly. One thing that changed how I build these systems: I stopped asking: "Will this workflow finish?" And started asking: "What state will the world be in when it finishes?" Because those are two very different questions. Most problems in long-running systems aren't caused by one big failure. They're caused by lots of small changes happening while the workflow is still alive. And if you don't account for that, eventually the workflow finishes successfully and still produces the wrong outcome. This is something we think about constantly

Introduction This is the second part of sharing my journey to create a microservices-based backend system, in the previous part, I introduced the system and its services. In this part I am going to mention the main things that happened with me from then until now. Transitioning to online DBaaS platforms The first main thing was to transition my databases from my local device to accessible over the internet. I used Neon and MongoDB Atlas . Although performance-wise it is better to deploy the database on the same server that the whole system so all services can access the databases without network overhead , This solution is better from a point, that is avoiding consuming the free VM instance that I got to host my main services, as with DBaaS platforms I will not require storage for databases or additional overhead to handle the DBMS on a free small resources-constrained VM. Observability Observability is one of the main concerns in any system, it gives the ability for the system to expose what happens inside it without any need to guess and manually trace the code to determine where the error may happened. I utilized OpenTelemetry , as I found it is most used and accepted tool to log, trace and collect metrics about the system and the traffic. Also, I loved that it is not related to specific framework, that is .NET by the way, I loved the idea that it is standalone tool. But logging and tracing and metrics collection will not be beneficial if we do not see it actually and can achieve monitoring from those telemetry data, so I used Grafana Cloud to export the telemetry data to some place that is accessible online and a tool that can generate dashboards and visualizing for the metrics ( Prometheus ), and UI that can show all the telemetry data easily. Deployment Until this point, we have a system that is fully functional and have consistent docker configuration for its services via Docker Compose, and it database is already there and available. Now this is the point at

The Problem: Feature Velocity Was Creating Structural Debt The system originally started as a simple feature delivery backend: A Django API powering agricultural insights Celery workers handling asynchronous processing Independent endpoints for each new capability A growing set of Earth Observation computations (NDVI, NDWI, etc.) At first, it worked. But as more features were added, a pattern emerged: Each feature introduced its own pipeline logic Observability was inconsistent across services API contracts drifted between frontend and backend Debugging required tracing multiple disconnected systems We weren’t scaling functionality. We were scaling fragmentation. The Turning Point: Features vs Platforms The key realization was simple: Features solve user problems. Platforms solve system problems. We were repeatedly rebuilding: Authentication flows Data ingestion logic Processing pipelines API validation layers Monitoring hooks Each feature was solving its own version of these concerns. That is where platform engineering became necessary. The Shift: Introducing a Platform Layer We introduced a platform layer between feature delivery and infrastructure. Instead of building isolated pipelines, we standardized: 1. Unified API Surface All Earth Observation workflows (NDVI, NDWI, and future indices) were normalized into a consistent API contract. Shared request/response structure Versioned endpoints Schema validation through serializers Central routing logic This eliminated endpoint fragmentation. 2. Standardized Processing Pipeline Celery tasks were refactored into a reusable pipeline pattern: Ingestion Validation Computation Storage Publishing Instead of feature-specific workers, we moved toward composable tasks. This allowed new indices or processing logic to plug into the same execution flow. 3. Observability as a First-Class Layer One of the biggest failures in the original system was visibility. We introduced: Structured logging across all services Traceable job IDs

I bombed a system design interview last year — not because I didn't know the architecture, but because I spent the first 5 minutes fighting Excalidraw. So I built SystemDesignBoard — a free, keyboard-first whiteboard specifically for system design interviews. What it does You open it, press a key, and start drawing. No account, no onboarding, no drag-from-a-sidebar friction. R → place a Service node C → place a Database/Cache/Queue A → connect two nodes N → open the scratchpad for scale math The features I'm most proud of Animated connectors that show communication type Instead of just drawing arrows, connectors visually encode how services talk: ⇄ sync — paired dashes (request + ACK) ≋ stream — near-solid fast line with glow (continuous pipeline) This matters in interviews — your interviewer can glance at your diagram and immediately understand the communication pattern. Cloud provider badges Tag any node as AWS (EC2, Lambda, RDS, S3), GCP (GKE, Cloud Run, Firestore), or Azure. Each subtype has its own icon. Trade-off logging Right-click any node → Log Trade-offs → attach your CAP theorem stance, consistency level, and scaling strategy directly to the component. Diagram-as-Code Type: [Mobile App] -> [API Gateway] [API Gateway] -> [Auth Service] [Auth Service] -> [Users DB] [Feed Service] -> [Posts DB x3] [Feed Service] -> [Redis Cache] Hit Apply — it auto-lays out the whole architecture in seconds. Export to animated GIF Export your diagram as a GIF that shows live traffic flow animations. Great for sharing after an interview or in a design doc. Tech stack React + TypeScript + Vite @xyflow/react (ReactFlow v12) for the canvas Zustand + Immer for state with full undo/redo html-to-image + gifshot for PNG/GIF export It's free and open No signup required. Works entirely in the browser. Free during beta. 👉 systemdesignboard.com Would love feedback — especially from anyone who's done system design interviews recently. What's missing? What's annoying? Drop a comment below