标签：#kafka

Atlassian details the Forge billing platform built for usage-based pricing across its cloud ecosystem. It processes large-scale usage events with correct attribution, deduplication, and aggregation using a streaming pipeline, idempotent processing, and layered storage to enable accurate billing, near real-time visibility, and reliable reconciliation across distributed services. By Leela Kumili

When a Spring Boot service needs to talk to another service without waiting on a synchronous HTTP call, message queues are the usual answer. Apache Kafka has become the default choice for this in most backend teams, but a lot of tutorials stop at a "hello world" producer and consumer that would never survive a real production load. Things like consumer retries, error handling, serialization of real objects, and graceful shutdown get skipped, and those are exactly the parts that page you at 2 a.m. In this tutorial, you will build a Spring Boot application that produces and consumes JSON messages over Kafka. You will configure a producer and a consumer, send a typed object instead of a plain string, handle deserialization errors so one bad message does not block your whole consumer group, and verify the whole thing works end to end. By the end, you will have a small but realistic messaging setup you can build on. Prerequisites To follow along, you will need: Java 17 or later installed. You can check your version by running java -version . A Spring Boot 3.x project. You can generate one at start.spring.io with the Spring for Apache Kafka dependency added. A running Kafka broker. The quickest way to get one locally is Docker, which the first step covers. Basic familiarity with Spring Boot, including how @Component and application.yml work. Step 1 — Running Kafka Locally with Docker Before writing any code, you need a broker to talk to. Running Kafka by hand involves Zookeeper, broker configuration, and a fair amount of setup, so you will use Docker Compose to bring up a single-broker cluster instead. Create a file named docker-compose.yml in your project root: services : kafka : image : apache/kafka:3.7.0 container_name : kafka ports : - " 9092:9092" environment : KAFKA_NODE_ID : 1 KAFKA_PROCESS_ROLES : broker,controller KAFKA_LISTENERS : PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS : PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES : CONTRO

Last year I was on a team that pushed 40 million events per day through Kafka. We had consumer lag alerts, rebalancing incidents, and a whole runbook for when the broker got behind. I understood how to operate Kafka. But I did not understand how Kafka works. So I built a tiny one. No dependencies. No Zookeeper. No JVM. Just Python and the core ideas. Here is what I learned. The Three Things Kafka Actually Does People say "Kafka is a message queue." That is not quite right. Kafka is a distributed commit log . It has three jobs: Accept writes from producers and append them to a log Let consumers read from any offset in that log Remember where each consumer group is up to That third one is the thing that makes Kafka different from a traditional queue. A queue forgets a message once it is consumed. Kafka remembers. You can replay. You can have 10 different consumer groups reading the same topic at different speeds. The code to implement this is smaller than you think. brokelite: A Message Broker in 120 Lines import threading import time from collections import defaultdict from typing import Dict , List , Tuple class Partition : """ Append-only log for one partition of a topic. """ def __init__ ( self ): self . _log : List [ Tuple [ int , bytes ]] = [] # (offset, message) self . _lock = threading . Lock () self . _next_offset = 0 def append ( self , message : bytes ) -> int : with self . _lock : offset = self . _next_offset self . _log . append (( offset , message )) self . _next_offset += 1 return offset def read_from ( self , offset : int , max_count : int = 100 ) -> List [ Tuple [ int , bytes ]]: with self . _lock : return [ ( off , msg ) for off , msg in self . _log if off >= offset ][: max_count ] def __len__ ( self ): return self . _next_offset class Topic : """ A topic is just N partitions. """ def __init__ ( self , name : str , num_partitions : int = 3 ): self . name = name self . partitions = [ Partition () for _ in range ( num_partitions )] def route ( self , k

A conversation between Uncle (a backend architect) and Nephew (a curious developer) about events, publishers, subscribers, Redis Pub/Sub, RabbitMQ, and Kafka The Beginning: What is an Event? 👦 Nephew: Uncle, I see words like "Redis Pub/Sub", "RabbitMQ", "Kafka" everywhere in job descriptions. What do they all do? Are they the same thing? 👨‍🦳 Uncle: (smiles) No, they're different. But before I confuse you with names, let me ask you something. Have you ever watched the news on TV? 👦 Nephew: Yes uncle, every morning! 👨‍🦳 Uncle: Perfect! So when the news channel says "Breaking News: India won the cricket match" - what happened? 👦 Nephew: Something important occurred... and they announced it! 👨‍🦳 Uncle: Exactly! That "something important occurred" is called an Event . In software, when something happens - like a user placing an order, a payment succeeding, or a file uploading - that's an event. 👦 Nephew: So event = something that happened? 👨‍🦳 Uncle: Yes! Think of it as news. When you place an order at Zomato, that's an event. When you get a payment notification from Google Pay, that's an event. When someone follows you on Instagram, that's an event. 👦 Nephew: Okay, I get it. But uncle, why is this "event" concept important? I can just write code directly, right? 👨‍🦳 Uncle: Ah! That's where the real story begins... The Problem: Spaghetti Code 👨‍🦳 Uncle: Imagine you own a food delivery company. A customer places an order. Now, what all needs to happen? 👦 Nephew: Umm... save the order, send email confirmation, alert the restaurant? 👨‍🦳 Uncle: Good! Let me write the code without events: function placeOrder ( orderId ) { saveOrder ( orderId ); sendEmailConfirmation ( orderId ); sendSMSNotification ( orderId ); notifyRestaurant ( orderId ); updateAnalyticsDashboard ( orderId ); addLoyaltyPoints ( orderId ); updateInventory ( orderId ); } Now, one day your boss says "Also send WhatsApp notification". What do you do? 👦 Nephew: Add another function call in the same code? 👨‍🦳 Unc

İlk yazıda Event Driven Architecture’ın temel kavramlarını, Kafka üzerinde topic/channel tasarımını, event-command ayrımını, schema contract’ları ve producer-consumer ilişkisini ele aldık. Bu yazıda odağı bir adım ileri taşıyıp event’in platform içindeki yaşam döngüsüne bakacağız. Çünkü EDA tasarımında asıl zorluk yalnızca event üretmek değildir. Asıl mesele, üretilen event’in güvenilir, izlenebilir, tekrar işlenebilir, zenginleştirilebilir ve farklı tüketiciler tarafından kullanılabilir hale gelmesidir. Bu yazıda şu sorulara odaklanacağız: Ham event platforma geldiğinde ne olur? Event nasıl doğrulanır, zenginleştirilir ve tüketilebilir hale gelir? Raw, validated, enriched ve curated topic’ler nasıl konumlandırılmalıdır? Bu yapı modern lakehouse mimarilerindeki Medallion yaklaşımıyla nasıl ilişkilendirilebilir? DLQ ve alert topic’leri ne zaman devreye girer? Replay, idempotency, monitoring, security ve governance nasıl düşünülmelidir? Event Pipeline Nedir? EDA mimarilerinde özellikle data platform projelerinde event’ler genellikle bir yaşam döngüsünden geçer. Bu yaşam döngüsü şöyle modellenebilir: raw -> validated -> enriched -> curated | | v v dlq alert Bu yapı, veri akışının aşama aşama olgunlaşmasını sağlar. Raw topic kaynaktan gelen ham event’i taşır. Validated topic schema ve temel kalite kontrollerinden geçmiş event’leri içerir. Enriched topic event’in referans veriler veya başka veri kaynaklarıyla zenginleştirilmiş halidir. Curated topic ise tüketiciler için güvenilir, normalize edilmiş ve iş anlamı netleşmiş event’leri temsil eder. Event Pipeline ve Medallion Architecture İlişkisi Bu yapı, modern lakehouse mimarilerinde sık kullanılan Medallion yaklaşımıyla doğal bir benzerlik taşır. Lakehouse tarafında Bronze katmanı ham veriyi, Silver katmanı temizlenmiş ve zenginleştirilmiş veriyi, Gold katmanı ise iş tüketimine hazır veri ürünlerini temsil eder. Kafka üzerindeki raw, validated, enriched ve curated topic’leri de benzer bir olgunlaşma mantığını akan veri ü

A survival guide for when everything goes wrong in production. There's a moment every engineer who works with Kafka experiences. You check the producer. Messages are sending. You check the consumer. Nothing. The consumer group shows zero lag because there's nothing to lag behind — as far as the consumer knows, the topic is empty. But it's not empty. The messages are there. Somewhere. In some partition, at some offset, behind some configuration you set six months ago and forgot about. Kafka doesn't lose messages. But it's very good at hiding them from you. Consumer Lag: The Number Everyone Watches Wrong Consumer lag is the difference between the latest offset in a partition and the offset your consumer group has committed. Simple concept. Dangerous in practice. The mistake: treating lag as a single number. Lag is per-partition. If you have 30 partitions and one consumer is stuck on partition 17 while the others are healthy, the total lag looks manageable. But partition 17's data is hours behind, and whatever downstream system depends on that data is serving stale results. Monitor lag per partition. Tools like Burrow, Kafka Exporter for Prometheus, or even kafka-consumer-groups.sh --describe break it down. If one partition's lag is growing while others are stable, you have a stuck consumer, a hot partition, or a poison message. A poison message is a record your consumer can't process — malformed data, unexpected schema, null where it shouldn't be null. The consumer throws an exception, the offset doesn't commit, and it retries the same message forever. Lag grows. The consumer looks "alive" because it's processing — just not making progress. The fix: dead letter queues. After N retries, move the message to a separate topic, commit the offset, and move on. Alert on the dead letter topic. Investigate later. Don't let one bad record block millions of good ones. Rebalance Storms: The Silent Killer Consumer rebalancing is Kafka's mechanism for redistributing partitions acro

This article explores Kafka's transition toward a cloud-native architecture, examining how tiered storage, FinOps telemetry, elastic consumer scaling, virtual clusters, and Share Groups reshape the operational and economic model of event streaming platforms. It also analyzes emerging diskless-storage proposals and their architectural trade-offs. By Viquar Khan

This week's Java roundup for May 18th, 2026, features news highlighting: GA releases of WildFly 40, Micronaut 5.0, Maven Embedded GlassFish Plugin 8.0 and Apache Fory 1.0; the May 2026 edition of Open Liberty; point releases of Gatherers4j, Apache and Kafka; and the seventh milestone release of Spring AI 2.0. By Michael Redlich

Schema proliferation builds slowly and gets expensive fast. One schema per event type feels right until there are ten tables, union queries spanning all of them, and a single field rename touching every schema. Discriminator-based schema consolidation collapses that to two tables, turning multi-table unions into a single query, while new variants are additive and don't break existing consumers. By Spoorthi Basu