标签：#STEM

It's easy to reach for the popular tool before asking what your system actually needs. For job queueing, the usual advice is to use RabbitMQ or Kafka. The underlying burden of using these tools will be additional processes to deploy, monitor and reason about. But what if using your existing database is possible? I built a job queueing system with a PostgreSQL database that cleared the bar without adding infrastructure. The next question will be, does this solution meet the criteria of a job queue? A job needs a few things to execute properly within a system. It needs to be persistent, surviving unforeseen crashes. Jobs must not be processed more than once by workers; one job should be processed once by one worker. Also, a job's state has to be tracked through every step. A job state must show when it's pending, completed or failed. That's the bar any solution must clear. With the Postgres approach, persistence comes free. Jobs live in a table so when a worker dies mid-job, the job still exists in a row in the db. Whereas with in-memory queues, a crash loses everything still in memory. A broker like RabbitMQ has to be configured for persistence and if configured wrongly, jobs get lost. A database however is fundamentally built for durability. Now, let's say three workers poll the queue at the same instant and run the same query. They'll see the same pending job at the top and nothing stops them all from grabbing it. If that job is a payment, the customer gets charged three times for one service. All the workers successfully process the job with no indication of an error or alerts. This is the requirement that seems to demand a real message broker, and it's exactly where people assume a database can't compete. It can. Postgres has a specific tool for exactly this. The SQL clause FOR UPDATE is used to lock rows. This can be called on a job when a worker picks it up to process. By default other workers will get blocked during this process, they'll wait for the lock to r

Originally published on nejckorasa.github.io . When a team moves from a monolith into microservices and event-driven, asynchronous systems, it inherits a class of problems that used to be someone else's: work that fails halfway through, steps that must not run twice, calls that return before the work is done. Temporal is a durable execution engine that handles a lot of this - you define a multi-step process, and it guarantees the process runs to completion even when workers crash in the middle. I've spent the better part of a decade building distributed systems in the money-movement core of banks - ledgers, payments, credit cards - a lot of it on Temporal, from short request-triggered workflows to ones that stayed open for weeks. This is the high-level guide I'd give a team making that jump: the principles worth internalising before you ship, not a full tutorial. Most of them aren't really about Temporal. They're the habits the async shift demands - Temporal just punishes you quickly when you skip one. Durable Execution: The Problem It Solves Distributed work fails in the middle. You call service A, it succeeds. You call B, it times out. The pod dies before C. Now you have half-finished work and no memory of how far you got. The usual fix is a pile of status columns, a cron job to find stuck rows, and retry logic hand-rolled for every step. Temporal's promise is that any process you start runs to the end. The runtime picture: there's a Temporal service (its own cluster), and your app runs worker processes that poll it and execute your code. As a workflow runs, Temporal records every step to an event history . If a worker dies, another picks the workflow up and replays that history to rebuild state, then carries on from where it left off, retrying anything that failed. The history is the source of truth, and it survives the crash. Most of the rules below fall out of that one fact. The Golden Rule: Workflows Decide, Activities Do There are two kinds of code in Tempora

From musical instruments and games to play pieces that encourage balance and dexterity, these toys will engage your child’s mind and body.

Building an operating system is a project I have had my eyes set on ever since I discovered free will in the realm of programming. Years ago, I did a reasonable amount of research, paying extra attention to the subject during my computer science degree and I was able to understand Operating System Theory and how it works from first principles but I never really got around to building one. I had only flimsy reasons for not embarking on it like "why build one when there are tons of working ones out there? The theoretical knowledge is enough" . More recently, I am ignoring the need to not re-invent the wheel for the joy of programming. So if you are interested in also rebuilding stuff because you can, join me on this series as I document how I am going to be building kluster. kluster is in its infancy and the direction is not clear but the one certain thing is that I will be building it entirely in Rust, save some assembly instructions and a linker script and I will be explaining every single line of code along the way. It will also be designed to target the raspberrypi 4 & 5, on qemu and on real hardware respectively. This is an opportunity for anyone who wants to see how Rust works at the lowest of levels to hop on and join the ride. Note that this series will be your biggest lesson on delayed gratification because we will write a lot of code before we even get to see anything meaningful on screen but I will foreshadow what you can get by the end of part 3 if you are patient enough: {{ image(src="/images/os-part3-result.png", alt="Part 3 Results OS Dev") }} You can also clone the source code for part 1 from Github and follow along. Project Setup First things first, let us setup the foundation of the project. I'll be straight with you, I love Rust and I enjoy using the Rust ecosystem in its entirety so I will stay true to that and use it as obsessively as any true Rustacean; I won't hold back. Without doubt, all the dependencies we need are freely available as long as

Back-of-the-envelope estimation for system design interviews Most people don't fail capacity math because the arithmetic is hard. They fail because they do it silently, produce a number they can't defend, and then never use it again for the rest of the interview. The math itself is trivial. The method is what's worth learning. Why interviewers ask Capacity estimation isn't a numeracy test. It's checking two things: Can you tell whether a design is physically possible before you commit to it? Do you know which constraint actually binds — storage, read throughput, write throughput, or bandwidth? A candidate who estimates 30,000 reads/sec and 200 writes/sec has learned something that changes the design. A candidate who computes petabytes of storage and then never mentions it again has just performed arithmetic. Round aggressively Precision is a trap. You're not producing a capacity plan; you're finding the order of magnitude. The single most useful substitution: 1 day = 86,400 seconds ≈ 10^5 seconds That's a 16% error and it makes every subsequent division doable in your head. Nobody will challenge it. Everyone will notice if you spend forty seconds long-dividing by 86,400. A few more worth having ready: 1 million requests/day ≈ 12/sec — round to 10 1 KB × 1 million = 1 GB 1 KB × 1 billion = 1 TB Peak traffic ≈ 2–3× average Replicated storage ≈ 3× raw ## Work in one direction Users → requests → QPS → storage → bandwidth. Don't jump around. Say each assumption out loud and label it as an assumption, so the interviewer can correct you early rather than watch you build on sand. A worked example Say we're designing a social feed. Given: 100M daily active users. Assumptions (stated, not smuggled in): Each user posts 0.2 times/day Each user reads their feed 10 times/day A post averages 1 KB including metadata A feed page shows 20 posts Writes 100M × 0.2 = 20M posts/day 20M / 10^5 = 200 writes/sec Peak (3×) = 600 writes/sec Reads 100M × 10 = 1B feed loads/day 1B / 10^5 = 10,0

This is going to be a very different article from what I usually write. No technical discussions, architecture deep dives, or engineering practices today. Instead, we're talking about something much older than software itself: music. We treat language like it's the default mode of human communication, like it's the real and only thing used to communicate, everything else is secondary, emotional, aesthetic, nice to have. But language is actually the outlier. It's the new protocol layered on top of something much older. Music is the original standard and we've basically forgotten how to read it. The Protocol Stack Think of communication like a network stack. Language is high-level. It's TCP/IP. Built on assumptions, needs learning, breaks the second you cross a boundary. You need: A shared vocabulary Syntactic understanding Cultural context Years of study if you actually want fluency It's powerful but It's also fragile. And it's recent . Written language is a few thousand years old. Spoken language is older, sure, but both are late abstractions compared to the hundreds of thousands of years humans have been syncing bodies to shared sound. Relative to that timeline? Language is yesterday's patch. Music? That's the lower-level protocol. The physical layer everything else runs on. A Japanese teenager at a Michael Jackson concert doesn't need to speak English. She doesn't need to understand what "Man in the Mirror" means as a concept. She also doesn't need a music degree. Music isn't zero -cost. Genre, culture, convention still shape how we hear it. But the entry barrier for emotional communication is way lower. A rhythm can hit urgency, celebration, sadness, or tension long before anyone understands the formal structure behind it. Her nervous system speaks that fluently. And so does everyone else in that stadium. How the Protocol Works Here's what happens when the song starts: 70,000 people stop being individuals and start being a distributed system synchronizing to the

Controller A records revocation epoch 12. Worker B, partitioned with a cached grant from epoch 11, starts another external action. The database is correct and the system is unsafe. Emergency stop is therefore a distributed protocol, not a Boolean field. What is verified In its July 21 disclosure, OpenAI says an internal benchmark used models with reduced cyber refusals and that a combination of models compromised Hugging Face infrastructure. The primary source is https://openai.com/index/hugging-face-model-evaluation-security-incident/ . Reporting on July 24 then described US discussion of emergency-shutdown and independent-audit proposals. The latter is policy coverage, not enacted law and not an extension of the official incident facts. Missing protocol details, impact boundaries, and remediation should remain unknown rather than inferred. Invariants and assumptions Assume workers, queue consumers, an authorization service, and external adapters can fail independently. Messages may be delayed, duplicated, or reordered; clocks have bounded error only if measured. Required invariants: No action starts with a grant epoch below the subject's revocation epoch. Cached grants expire within a declared lease bound. Restart cannot lower a persisted epoch. Duplicate revocation converges to the same or higher epoch. Completion means every registered executor acknowledged or its lease expired. revoke(subject, epoch=13) -> durable CAS max(current, 13) -> publish {subject, epoch:13} -> executors persist max(local, 13), ack -> controller waits for ack set OR lease expiry -> issue completion receipt with missing/expired members Failure injection Property Acceptance rule delay revocation event lease bounds stale authority no start after local lease expiry duplicate epoch 13 idempotence epoch remains 13+ deliver 13 before 12 monotonicity never returns to 12 worker restarts durability loads persisted epoch before work controller partition fail closed no new lease after expiry A minim

I cranked Apple’s iOS 27 beta to maximum transparency on my iPhone and enjoyed every warped second.

8 min read · telecom provisioning platform, 30M+ subscribers Most companies avoid migrating their monolith for one reason: they imagine it as a single, terrifying event — months of a feature freeze, a weekend cutover, and a rollback plan that's really just hope. That fear is reasonable. A big-bang rewrite of a live system serving 30M+ subscribers really would be terrifying. So we didn't do that. We used a pattern that lets you migrate a monolith one slice at a time, while the system keeps running and the product team keeps shipping features — the same pattern Martin Fowler named Strangler Fig , after the vine that grows around a host tree, gradually taking over, until eventually the original tree is no longer needed. Why the "just rewrite it" instinct is usually wrong The instinct to rewrite a legacy monolith from scratch is understandable — the old code is scary, undocumented, and nobody wants to touch it. But a full rewrite has a well-known failure pattern: it takes far longer than estimated, the business can't freeze feature development for that long, and by the time the rewrite is "done," the old system has changed underneath it and the rewrite is already out of date. The alternative isn't "don't migrate." It's: migrate in slices small enough that each one is boring , and never require the business to stop shipping while you do it. The pattern: a facade, and one slice at a time Strangler Fig works by putting a routing layer — a facade or API gateway — in front of the monolith. At first, 100% of traffic passes through to the old system untouched. Then, one capability at a time: Build the new version of that one capability as an independent service. Update the facade to route just that capability's traffic to the new service. Run both in parallel long enough to trust the new one (see "shadow traffic" below). Retire that piece of the old monolith. Repeat for the next capability. At every point in this process, the system is fully functional. There's no "half-migrat

Your Contact Form 7 form can work perfectly from a user's perspective and still fail to deliver a lead to your CRM. The visitor fills out the form. The browser shows a success message. The WordPress form appears to have submitted correctly. But when you open Systeme CRM, the contact is nowhere to be found. This is one of the most confusing problems in form-to-CRM integrations because a successful form submission does not necessarily mean a successful API request. The complete workflow has multiple stages: Visitor ↓ Contact Form 7 ↓ WordPress ↓ API Request ↓ Systeme CRM ↓ Contact Record ↓ CRM Automation A failure at any stage can break the workflow. The key to debugging the integration is to stop treating the form submission as a single event and start checking each stage separately. First, separate the two different types of success There are two different questions: Did Contact Form 7 submit the form? This is a WordPress-side question. Did Systeme CRM accept and process the API request? This is an API and CRM-side question. These are not the same thing. A form can successfully collect: Name: John Doe Email: john@example.com Company: Example Inc. while the API request fails because: the endpoint is incorrect authentication is missing the request method is wrong the JSON payload is invalid the CRM expects different field names a required field is missing The first debugging step is therefore to identify exactly where the data flow stops. Step 1: Confirm that Contact Form 7 is collecting the expected data Start at the beginning. Look at the form fields: [text* your-name] [email* your-email] [tel your-phone] [text company] [textarea your-message] The important values are the actual field names: your-name your-email your-phone company your-message A common mistake is to assume that the visible label is the field name. For example: Visible label: Full Name Field name: your-name The integration needs the submitted field value associated with the actual field name. Before

When a new video shows up in one of our regional crawlers, three things need to happen almost immediately: the SQLite FTS5 search index for that region needs a new row, the discovery ranking cache needs to be invalidated, and the sitemap generator needs to know a URL was born. For a long time we did all of this inline, inside the same cron process that fetched the video. It worked until it didn't. A slow FTS5 rebuild would stall the fetch loop, a sitemap write would fail silently, and a crash halfway through meant one region had the video indexed and another didn't. The fetch and the fan-out were fused together, and every failure was a partial failure. The fix was to stop treating "a video was discovered" as a function call and start treating it as an event. At TrendVidStream we run discovery across 8 regions, and the moment we introduced NATS JetStream as the spine between the crawler and the downstream consumers, the whole system got calmer. This post is the concrete version of how we did it: the stream config, the publishers, the consumers, and the mistakes we made that you can skip. Why Not Just Use a Queue Table in SQLite We already had SQLite everywhere, so the obvious move was a jobs table. We tried it. The problems showed up fast: Polling latency vs. load tradeoff. Poll every second and you hammer the DB with mostly-empty SELECT queries across 8 regions. Poll every 30 seconds and your search index lags noticeably behind your crawler. No fan-out. One row, one worker. If the sitemap generator and the FTS5 indexer both need the same event, you either duplicate rows or invent a consumed_by bitmask. Both are ugly. Locking. SQLite's writer lock means the queue table and the actual data table start contending under the multi-region cron bursts we run. Cross-region delivery. Our regions aren't all on the same box. A queue table doesn't cross machines without you building a replication story on top. JetStream solves all four: push-based delivery (no polling), multipl

1. Subsystem Decomposition 1.1 The Decomposition Problem A subsystem decomposes a codebase into smaller, cohesive units. Two primary axes of decomposition exist: Technical axis : grouping by component type (controller, service, model, view) Functional axis : grouping by business capability (cataloguing, circulation, etc.) 1.2 Tension Between Framework Prescriptions and Decomposition Strategy Organizing top-level subsystems functionally may create friction with frameworks that prescribe a technical-first structure. Concrete examples: Rails enforces model, view, and controller directories at the root level, making functional decomposition awkward without additional mechanisms like Rails Engines Sinatra (a microframework) imposes minimal structure, leaving architectural decisions entirely to the team Frameworks with rigid prescriptions constrain architectural choices. Frameworks with no structure shift the entire burden onto the team with no guidance. This second approach might be fine for teams that know what they are doing and how to shape the architecture properly. Not everyone needs guidance from the framework. 1.3 Contexts as a Middle Ground Phoenix provides contexts as a compromise: Explicit, guideline-oriented subsystems that enable functional decomposition without rigid enforcement Contexts define functional boundaries while allowing technical organization to remain nested within them Functional blocks may later evolve into microservices, but this is optional The same decomposition serves equally well in a modular monolith or a distributed architecture The choice depends on team needs, scaling requirements, and operational maturity, not on the decomposition strategy itself 2. Pipeline Topology and Data Flow 2.1 The Unix Pipeline Model Unix pipelines model data flow through a single stream connecting stdout to stdin. This forms a linear chain where each stage's output becomes the next stage's input. Key characteristics: Each stage has exactly one input and one o

The pricing formulas in Motor, the estimating engine I built for a water feature shop, did not come from the manual. I pulled 32 of them out of the JavaScript behind Aquascape's contractor calculator, the tool contractors actually use to bid jobs. The manual was sitting right there, official and free. Ignoring it was the best design decision in the whole system. A vendor never ships a sloppy calculator Why trust the calculator over the manual? Because of what happens when each one is wrong. If the manual sizes a pump wrong, a reader shrugs and moves on. If the calculator sizes a pump wrong, a contractor bids a job at that number, wins it, and loses money on the install. Then the phone rings. So calculators get fixed and manuals drift. Give it ten years and the two quietly disagree, and everyone in the trade knows which one to trust without anyone saying so. A vendor will ship a sloppy PDF. They will never ship a sloppy calculator. Documentation is what a domain says about itself. The artifacts money flows through are what it actually believes. Once you see that split, you cannot stop seeing it. The other half was in old invoices Formulas only get you to cost. What a shop charges on top of cost is a belief about its market, and no vendor document holds that number. So I pulled 132 historical quotes out of the shop's CRM. Real quotes, sent to real customers, most of them paid. I calibrated Motor's markup against those, then checked its output against what the shop had actually charged. The result: Calibrated against 132 real quotes, Motor's estimates landed within 5 percent of what the shop actually charged, with no pricing rule taken from documentation. I could have just asked the owner what his markup was. But what an owner says and what his invoices show are rarely the same number, and the invoices are the ones customers paid. When the two disagree, believe the invoices. The same bug in a different industry I build and run systems in several industries, and the sur

Every engineer eventually hits this phase: “My design looks okay… but something feels off.” No compile errors. No obvious bugs. But still: responsibilities feel scattered services feel too big entities feel too thin logic feels duplicated boundaries feel unclear This is normal. Because domain modeling is not about getting it right in one attempt. It is about refining structure until the business behavior becomes clear. Step 1 — Start With the Symptom, Not the Code If your design feels wrong, don’t immediately rewrite everything. First identify the symptom: Common symptoms: too many “Manager” services logic repeated in multiple places unclear ownership of rules too many dependencies between modules frequent “if-else explosion” Each symptom points to a specific modeling issue. Step 2 — Check If Invariants Are Scattered Ask: “Where are my business rules living?” Bad sign: Rules inside services + controllers + helpers This leads to: inconsistent behavior duplicated validation broken business guarantees Good design: invariants live close to the entity or aggregate root Step 3 — Check Entity vs Service Confusion A very common issue: Entities become dumb: only fields no behavior Services become overloaded: all logic all rules all decisions This creates: Anemic Domain Model + Fat Services Fix mindset: Entity = owns behavior + protects state Service = coordinates workflows Step 4 — Check Your Aggregate Boundaries Ask: “What must stay consistent together?” If your answer is unclear, you likely have: wrong aggregates or missing aggregates Example problem: Cart and Order sharing logic This causes: inconsistent pricing unclear lifecycle ownership Fix: Cart = intent Order = truth Step 5 — Look for “Hidden Coupling” Hidden coupling happens when: one module depends on internal state of another multiple services modify same data business rules are duplicated across boundaries This leads to fragile systems. Strong design ensures: each domain owns its own truth. Step 6 — Validate Stat

Before a young Athenian took his full place in the city, he spent two years in the ephebeia. Training happened in the gymnasion, organized by tribe, the same tribes that would later send him to represent them in the Boule itself. Nobody handed him full standing first and hoped the judgment would follow. This should have been post seven. It’s showing up as sixteen because the gap only became visible once the room was real enough to test against. The Agora described what a Spec Session does. It never asked whether everyone walking into that room shares an accurate picture of what the agent can actually do. Most rooms don’t. Someone watched a demo and thinks the agent can do anything. Someone else got burned by a bad output three weeks ago and doesn’t trust it with anything real. Nobody’s intuition has been tested against the same tasks, and the spec that comes out of that room ends up too ambitious or too conservative depending on whose untested belief happened to speak first. That gap has a name in Athens. The gymnasion existed because nobody was handed a place in the city first and expected to develop judgment on the job. The ephebeia ran two years, training built around a specific fact. Physical readiness and civic judgment weren’t taught in separate places. They happened in the same space, under the same supervisors, organized by the same tribal groupings that would later structure how the city actually governed itself. The same word, gymnasion, ended up naming both the training ground for eighteen-year-olds and the buildings where Plato and Aristotle did their most serious thinking. Academy and Lyceum were gymnasia first. Nobody separated the trial from the reflection. The trial was how the reflection got earned. That’s the part worth taking seriously. Testing a tool and understanding a tool were never two different activities. The testing is how the understanding gets built. A team that reads documentation about what an agent can do has a description. A team tha

In this article, we're going to explore Event Schema evolution with Event versioning 10. Event Schemas Will Eventually Change No event schema stays the same forever. As businesses grow, regulations shift, products gain new features, and processes become more complex, the data shared between services must evolve as well. This evolution is not optional—it is a natural consequence of a system adapting to changing requirements. Many teams initially assume they can simply update an event whenever needed. This assumption may hold when there is only one producer and one consumer, but real-world systems rarely remain that simple. Over time, multiple consumers emerge, each with its own responsibilities and release cycles. A typical system often looks like this: OrderConfirmed | +------------------+-------------------+ | | | v v v Inventory Billing Notification | v Analytics | v Customer Insights Each consumer evolves independently. Some services may deploy updates weekly, while others might release changes quarterly. In some cases, consumers may even belong to external teams with entirely different priorities and timelines. Because of this, producers cannot assume that all consumers will upgrade simultaneously. Schema evolution, therefore, is not just about modifying data structures. It is fundamentally about maintaining compatibility across independently evolving systems. Compatibility Is More Important Than Version Numbers When discussing schema evolution, teams often focus immediately on versioning. While versioning is useful, compatibility is far more critical. Without compatibility, versioning alone cannot prevent system breakage. Consider the following event: { "orderId" : "ORD-1001" , "customerId" : "CUS-501" , "totalAmount" : 249.99 } Now imagine a new requirement introduces currency. One approach might replace the existing field entirely: { "orderId" : "ORD-1001" , "customerId" : "CUS-501" , "amount" : { "value" : 249.99 , "currency" : "USD" } } Although the data mo

I want to start with a moment most of us have lived through. It's 3 a.m. A dashboard is red. You're eight terminals deep in grep , trying to work out which service actually fell over and why. And the whole time there's this nagging feeling that you're doing archaeology on a system you wrote last month. Here's what got under my skin about it. The answer was never actually lost. Back in the source code it said, in plain terms, that the payment service talks to Postgres through a connection pool. That this retry backs off three times. That this particular dependency is external and you must never, ever try to "just restart it." That was all right there at build time. Then we packaged everything up, deployed, threw that structure in the bin, and asked a sleep-deprived human to reconstruct it from log lines. That gap bugged me enough that I spent a while building something around it. This post is about that. Autoscaling is good at the wrong problem We've gotten genuinely good at reacting to resource pressure. Traffic climbs, a box gets slow, CPU pins, and the autoscaler adds capacity or sheds load. No complaints there, it's kept things running for years. The problem is it has no idea what your app is for . It can't tell a service that's slow because it's healthy and hammered from a service that's fast because it's quietly writing garbage to the database. It never had a model of the application in the first place. So a whole category of failures just sails right past it. A connection pool getting drained by something downstream. A poison message kicking off a retry storm. A schema change that breaks one code path and leaves the other one looking perfectly fine. Infrastructure that only thinks in CPU and memory is blind to all of that. The "throw an LLM at it" era The going answer right now is to bolt a large language model onto your observability stack. Fire hose all the logs, traces, and metrics at a big central model and ask it what happened. I get why. I also think it'

Metron was the Greek word for a measure: the standard you judge a thing against. It gives us metric, and metronome. Pick the wrong metron and everything you count against it comes out wrong, no matter how carefully you count. Ask most teams how they know AI adoption is working, and the answer is some version of: we’re shipping more. More PRs, more tickets closed, more velocity on the board. None of those numbers were ever measuring what people thought they were measuring. They were proxies, and bad ones, for whether the team was creating value. AI didn’t break that. It just made the proxies lie louder. An agent can produce PRs faster than any team can review them. It can close tickets all day. None of that tells you whether the thing built was worth building, or whether it fixed the actual problem, or whether anyone downstream is better off. Burning tokens isn’t the same thing as creating value. It just looks the same on a dashboard built to reward the first thing. The same DORA numbers I pointed at earlier are blunt about it: individual output climbs while delivery throughput and stability drop. Same bottleneck, counted at the wrong end . This is Theory of Constraints in one sentence: speeding up a step that wasn’t the bottleneck doesn’t speed up the system. It just piles more work in front of whatever the real bottleneck is. Implementation used to be the constraint. Now it isn’t. Review is. Coordination is. Making sure everyone agrees on what “done” even means is. Agents made the fast part faster and left the slow part exactly where it was, except now it’s buried under more output arriving to be reviewed by fewer people who understand any of it. Measuring individual speed after that shift is measuring the wrong clock. The team can be faster and worse off in the same quarter. The instinct, when this gets noticed, is to add another skill to the agent. Automate the review step too. Automate the coordination. Whatever’s slow, throw a capability at it. That doesn’t fix

Xona aims to deploy 258 satellites into low-Earth orbit as a GPS alternative.

"We're using Kafka with exactly-once semantics, so we don't have to worry about duplicates." I've heard this in architecture reviews, design docs, and postmortem explanations. It represents a misunderstanding of what Kafka's exactly-once guarantee actually covers, and the gap between the promise and the reality has caused real production incidents. What Kafka's Exactly-Once Actually Covers Kafka's exactly-once semantics (EOS), introduced in 0.11.0, operates at two levels: Producer idempotence ( enable.idempotence=true ): The producer assigns each message a sequence number. The broker deduplicates messages with the same producer ID and sequence number. This prevents duplicates caused by producer retries — the message lands in the Kafka partition exactly once, regardless of retry count. Transactions ( transactional.id ): Allows a producer to write to multiple partitions atomically. Either all writes commit or none do. Combined with isolation.level=read_committed on consumers, readers only see committed transactions. Together, these give you exactly-once message delivery within the Kafka cluster. What Exactly-Once Does Not Cover Here's the boundary that engineers miss: Kafka's exactly-once guarantee is scoped to the Kafka cluster. The moment your consumer does anything outside Kafka — writes to a database, calls a REST API, publishes to a cloud queue — you're outside the transaction boundary. Consider a typical consumer: consumer . poll ( records ); for ( record : records ) { database . save ( process ( record )); // External write — outside Kafka transaction } consumer . commitSync (); If the application crashes after database.save() but before commitSync() , Kafka re-delivers the message. The consumer reprocesses it. The database now has two writes for the same event. Enabling producer idempotence on the consumer's Kafka writes does not fix this. The Patterns That Actually Give You End-to-End Safety Idempotent Consumers Design consumer processing logic to be idempote