标签：#datapipeline

What Rebuilding Kafka From Scratch Actually Teaches You There's a gap between using a system and understanding it. Most engineers never close that gap, and honestly, most of the time that's fine. Kafka works. Topics, producers, consumers, pull the levers, ship the data. Done. But then you hit a weird latency spike, or a consumer group stalls in a way that doesn't match the docs, or replication starts behaving like it has feelings. And suddenly "I know the terminology" doesn't cut it anymore. That's exactly why this rebuild post is worth your time. The Abstraction Tax Every framework you use charges you an abstraction tax. The tax isn't the dependency. It's the mental model debt you carry when something goes wrong and you don't know what layer to blame. Kafka's tax is particularly sneaky because its concepts sound simple: topics are channels, partitions are buckets, offsets are counters. You can get productive fast. And then that simplicity starts lying to you. Why does lag spike when throughput looks fine? Why does adding consumers past the partition count do nothing? Why does a rebalance tank your throughput for 30 seconds? These aren't Kafka quirks. They're direct consequences of how the log is actually structured, consequences that become obvious the second you implement it yourself. What the Rebuild Exposes When you write the log append yourself, the offset model stops being abstract. An offset isn't just a cursor, it's a byte position in a segment file. Consumers aren't "reading from a partition," they're replaying a structured log from a known position. Replication isn't a background checkbox, it's a follower explicitly fetching and acknowledging write positions. A few things that tend to click when you go through this kind of exercise: Segment files and retention , Kafka doesn't delete old messages by scanning. It deletes whole segment files once they're past the retention boundary. If you've ever been surprised by how Kafka handles disk, this is why. Why par

Most teams don't consciously choose their messaging infrastructure. They inherit it. Someone used Service Bus on the last project, it worked fine, and now it's the default answer for every async communication problem that comes up. Two years later, you're bending it into shapes it was never designed for, and the operational pain gets blamed on "distributed systems being hard" rather than on the actual culprit: a tool being asked to do a job it doesn't fit. The problem isn't that Service Bus, Event Grid, or Kafka are bad. It's that they solve genuinely different problems, and conflating them doesn't just create technical debt — it creates architectural liability that compounds over time. The Real Difference Is the Communication Contract, Not the Feature List When you put these three tools side by side in a comparison table, you'll find overlapping columns. All three move messages between systems. All three have some delivery guarantee story. That's where the surface-level comparison breaks down and people make bad decisions. The more useful question is: what contract does your system need to uphold with the data it moves? Service Bus is fundamentally about reliable, ordered processing with strong delivery guarantees. It's designed for the case where every message matters individually, where you need competing consumers pulling from a queue, where poison message handling and dead-lettering are first-class concerns. If you're coordinating business process steps or handling financial transactions where exactly-once semantics matter, this is the right shape of tool. Event Grid is about reactive routing. Something happened in your infrastructure or your application, and you want other things to respond to it. It's push-based, fan-out-friendly, and optimized for low-latency notification rather than high-volume throughput. It's not trying to be a buffer. If you're triggering downstream workflows in response to blob uploads, resource state changes, or custom application even