标签：#service

Standard chaos engineering assumes experiments stop cleanly, blast radius is knowable in advance, and production is fair game. Payment systems violate all three. Salim Adedeji describes ECS-specific failure modes from enterprise deployments: a 60-second DNS TTL that produced 93-second failover, retry logic amplifying database load 2.4x, and AZ rebalancing loops that generic tooling misses. By Salim Adedeji

Scaling Kafka Consumers in Spring Boot: How We Cut Lag and Saved Latency When scaling high-throughput event-driven microservices in fintech, default Spring Kafka consumer configurations often run into throughput limits under peak loads. Here is the exact production setup we engineered to resolve consumer lag and reduce API processing latency by 35%. 1. Concurrency Tuning Over Single-Threaded Listeners By default, @KafkaListener operates with concurrency = 1. When a partition receives high message volume, processing gets backlogged. @Configuration @EnableKafka public class KafkaConsumerConfig { @Bean public ConcurrentKafkaListenerContainerFactory < String , PaymentEvent > kafkaListenerContainerFactory ( ConsumerFactory < String , PaymentEvent > consumerFactory ) { ConcurrentKafkaListenerContainerFactory < String , PaymentEvent > factory = new ConcurrentKafkaListenerContainerFactory <>(); factory . setConsumerFactory ( consumerFactory ); factory . setConcurrency ( 6 ); // Matches number of partition splits factory . getContainerProperties (). setAckMode ( ContainerProperties . AckMode . MANUAL_IMMEDIATE ); return factory ; } } 2. Explicit Batch Processing and Idempotency Instead of committing offset per message, processing batches with manual acknowledgments ensures atomic handling: @Service public class PaymentEventConsumer { @KafkaListener ( topics = "payment.settlement.v1" , containerFactory = "kafkaListenerContainerFactory" ) public void consume ( ConsumerRecord < String , PaymentEvent > record , Acknowledgment ack ) { try { processPayment ( record . value ()); ack . acknowledge (); } catch ( Exception ex ) { log . error ( "Failed processing record key: {}" , record . key (), ex ); // Route to Dead Letter Queue (DLQ) handleDeadLetter ( record ); ack . acknowledge (); } } } 3. Key Takeaway Scaling Kafka consumer pipelines requires matching topic partition count with container concurrency, tuning database connection pools and implementing dead letter queues for fail

Domain-Driven Design for Microservices: Building Scalable Systems with Clear Domain Boundaries Introduction Domain-Driven Design (DDD) isn't just another architecture pattern—it's a philosophy that aligns technical decisions with business reality. When building microservices at scale, DDD becomes essential. Without it, you end up with services that don't respect business domains, unclear responsibilities, and integration nightmares. Why DDD Matters for Microservices Microservices force you to make decisions about boundaries. The question isn't whether you'll decompose your system—it's whether you'll do it thoughtfully using DDD principles, or accidentally create distributed monoliths. DDD answers three critical questions: Where should a service boundary exist? (Bounded Contexts) How do we communicate across services without coupling? (Domain Events, Anti-Corruption Layers) How do distributed teams understand the same problem? (Ubiquitous Language) Core Concept 1: Bounded Contexts A Bounded Context is a boundary within which a domain model is applicable. Each microservice should typically map to one or more Bounded Contexts. Java Example: E-commerce System // Ordering Context - Bounded Context 1 public class Order { private String orderId ; private List < OrderLineItem > lineItems ; private OrderStatus status ; // PENDING, CONFIRMED, SHIPPED, DELIVERED private LocalDateTime createdAt ; public void confirmOrder () { if ( this . status != OrderStatus . PENDING ) { throw new InvalidOrderStatusException ( "Cannot confirm non-pending order" ); } this . status = OrderStatus . CONFIRMED ; } } // Inventory Context - Bounded Context 2 public class InventoryItem { private String skuId ; private Integer availableQuantity ; private Integer reservedQuantity ; public void reserveStock ( Integer quantity ) { if ( availableQuantity < quantity ) { throw new InsufficientStockException ( "Not enough stock to reserve" ); } this . reservedQuantity += quantity ; this . availableQuantity -

Uber’s GitFarm provides Git operations as a centralized service, eliminating local repository clones across large scale monorepo workloads. The platform uses prewarmed checkouts, ephemeral sandboxes, repository synchronization, and gRPC streaming to reduce resource consumption and startup latency for automation services operating across thousands of repositories. By Leela Kumili

Most gateway tutorials stop at "here's how you route a request." That's the easy 20%. The hard part is what happens when a client hammers you with requests, a downstream service falls over mid-traffic, or you're staring at a 500 trying to figure out which of your four services actually caused it. I wanted to build something that hits those problems on purpose, so I put together spring-gateway-sample : a public gateway , an api-server that fans out to two downstream services, and a full observability stack sitting behind all of it. It's not a real product and never will be. But I tried to make it behave like one — including the annoying bits, like config tradeoffs and races that most demos just quietly ignore. Stack, for context: Spring Boot 4.1, Spring Cloud Gateway on WebFlux, Resilience4j, Redis, Postgres, Keycloak, Prometheus/Grafana/Tempo/Loki, and a small Vue 3 app for throwing traffic at it from a browser. The system, in one request Browser (Vue traffic simulator) │ Keycloak PKCE login + API key ▼ Gateway ── JWT + API-key auth, Redis rate limiting ──▶ routes to │ ▼ api-server ── WebClient delegation, circuit breakers, Caffeine cache ──▶ │ │ ▼ ▼ product-service pricing-service (JPA / Postgres) (JPA / Postgres) Every hop re-validates the JWT on its own — defense in depth, so the gateway isn't the single thing standing between the internet and the data. The gateway also checks an API key on top, because a JWT tells you who the user is, not which client application is calling on their behalf. You need that second identity if you want per-client rate limits or the ability to revoke one app's access without touching anyone else's. Two checks, one specific order Every request needs a Keycloak JWT and an API key, and the order they're checked in isn't an accident: Missing or expired JWT → 401 , before the API key is even looked at. Valid JWT, bad API key → 401 , but a different error code. Both valid, wrong role → 403 . Why bother with the ordering? Because "you're no

California residents have a legal right to access the data that companies collect about them. Actually exercising that right is a burdensome nightmare.

Standardized driver interface aims to let devices talk to AI and each other.

HelloFresh’s organic meal kit Green Chef offers transparent sourcing, layered cooking, and trustworthy gluten-free dishes.

When you're building an AI evaluation platform with multiple microservices, the "core" services get all the attention — the evaluation engine, the scoring system, the RAG pipeline. But a platform doesn't work without the connective tissue: the workflow orchestration that keeps humans in the loop, the taxonomy engine that classifies tasks intelligently, the platform service that ties authentication together, and the evaluation suites that ensure models actually remember context. These four services don't make headlines, but they're what turned a collection of microservices into an actual platform. Here's what went into each one and why the engineering decisions mattered. Workflow Orchestration: The Human-in-the-Loop Engine AI evaluation is not fully automated — and it shouldn't be. Certain decisions require human judgment: Is this model response harmful? Does this evaluation rubric make sense for this domain? Is this edge case a genuine failure or acceptable behavior? The workflow orchestrator manages these decision points. It coordinates multi-step evaluation workflows where some steps are automated (LLM scoring, data validation) and others require human approval before the pipeline continues. The Architecture The core is a state machine built on FastAPI and PostgreSQL. Each workflow is a DAG (directed acyclic graph) of tasks, where each node can be: Automated: Runs immediately, calls another service (scoring, data enrichment), stores the result Human gate: Pauses the workflow, notifies the assigned reviewer via the notification service, waits for approval/rejection Conditional: Routes to different branches based on previous step outcomes (e.g., if confidence score < threshold, escalate to senior reviewer) State transitions are persisted in PostgreSQL with Alembic-managed migrations. Every transition is logged — who approved what, when, and with what context. This audit trail turned out to be critical for client reporting. Real-Time Updates with WebSocket The origin

The cyberattack at CareCloud resulted in one of the largest reported data breaches in the U.S. healthcare industry this year.

Sam Newman discusses the concept of progressive collapse in civil engineering and how it applies to distributed systems. Using real-world examples - from the 1968 Ronan Point tower failure to AWS outages - he shares crucial resilience engineering strategies for software leaders. Learn how to strengthen components, isolate failures, and reduce interconnections to prevent catastrophic cascades. By Sam Newman

Amazon Web Services has extended Amazon Bedrock AgentCore with runtime instances, a new compute option that gives AI agents persistent infrastructure purpose-built for complex long-running workflows and multi-agent coordination. By Matt Saunders

With the Azure Kubernetes Services (AKS) platform, containerized workloads can be efficiently managed and scaled. However, the full potential of AKS is only realized when it is seamlessly integrated with other Azure services. This enables a complete cloud-native environment that is scalable, secure, and automatable, while providing maximum flexibility. In this final post of the series, we will show you how AKS can be integrated with other Azure services to create a robust and holistic platform for your applications. Why Integrating AKS into the Azure Cloud Is Crucial AKS provides a highly available and scalable infrastructure for managing containerized applications. However, integrating it with other Azure services like Azure DevOps, Azure Monitor, Azure Active Directory (Entra ID), and Azure Storage extends functionality and optimizes workload management. By leveraging Azure services alongside AKS, companies can: Ensure enhanced security for their containerized applications. Build robust monitoring and logging solutions to monitor the state of applications at all times. Set up automated pipelines for deployment and scaling. Seamlessly exchange data and status information across various Azure services. Key Azure Services to Integrate with Your AKS Platform Azure Active Directory (AAD) for Authentication and Security Azure Active Directory (AAD) provides comprehensive identity and access management that can be directly integrated with AKS. This ensures that only authorized users and services can access your Kubernetes clusters. With Azure RBAC (Role-Based Access Control), you can define granular access permissions for different users and teams, increasing the security of your environment. AAD Pod Managed Identities enable your AKS applications to securely access Azure resources like Azure Key Vault or Azure Storage without the need to manually manage sensitive credentials. Azure DevOps for CI/CD Pipelines Azure DevOps is one of the best solutions for automating CI/CD

The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap. Then you load-test it. I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned. What a correct implementation looks like Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker: @Scheduled ( fixedDelay = 5000 ) @Transactional public void publishPendingEvents () { List < OutboxEvent > batch = outboxRepository . findTop20ByStatusOrderByCreatedAtAsc ( OutboxStatus . PENDING ); for ( OutboxEvent event : batch ) { event . setStatus ( OutboxStatus . PROCESSING ); outboxRepository . save ( event ); try { kafkaTemplate . send ( event . getTopic (), event . getPayload ()). get (); event . setStatus ( OutboxStatus . PUBLISHED ); } catch ( Exception e ) { event . incrementRetryCount (); if ( event . getRetryCount () >= MAX_RETRIES ) { event . setStatus ( OutboxStatus . FAILED ); } else { event . setStatus ( OutboxStatus . PENDING ); } } outboxRepository . save ( event ); } } This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed. Here is what that correctness does not cover. Gap 1: Your throughput ceiling is a config line fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycl

I’ve tried most of the ready-to-eat meal delivery services in the country. CookUnity is the one that feels like real cooking, from real chefs.

One of the most common assumptions in engineering leadership is that a rising number of reported incidents signals declining system reliability. However, a recent article from Great Circle argues that the opposite is often true: an increase in incident counts may actually indicate that an organization's incident management culture is improving. By Craig Risi

n today's world of cloud-native development, businesses require powerful, scalable, and flexible platforms that help developers efficiently build and operate their applications. An Internal Developer Platform (IDP) based on Azure Kubernetes Services (AKS) provides an optimized environment that brings together all the key components for modern software engineering. This article explains how to develop such a platform using AKS, what key components are required, and how to integrate them optimally. What is an Internal Developer Platform (IDP)? An internal developer platform is a set of tools, processes, and automations provided to developers to simplify the entire software development process. It offers a standardized environment where developers can write, test, and deploy code without worrying about the infrastructure or underlying complexities. An IDP built on Azure Kubernetes Services (AKS) also allows for the operation of containerized applications in a fully managed, highly available, and scalable environment. Core Components of a Development Platform on AKS When building an internal developer platform based on AKS, several key components ensure an efficient and robust system. Here are the essential elements: Azure Kubernetes Services (AKS) as the Central Platform AKS forms the core of the development platform. It provides a scalable and managed Kubernetes environment where all containerized applications run. With full integration into other Azure services, developers can access a wide range of tools to efficiently manage, monitor, and scale their workloads. Service Mesh for Managing Microservices Communication In a microservices architecture, which is commonly used in modern cloud-native applications, communication between services plays a crucial role. A Service Mesh like Istio or Linkerd enables the management and monitoring of this communication. It provides features such as load balancing, traffic management, security policies, and monitoring for microservi

Netflix has described how it redesigned the streaming pipeline behind Service Topology, its real-time service dependencies map, to support production scale. The system uses three stages to separate intermediary resolution from enrichment and persistence, propagates backpressure to Kafka rather than dropping records, and uses server-sent events instead of gRPC for high-volume internal transfers. By Eran Stiller

Digital transformation has led companies to organize their infrastructure and development processes in entirely new ways. To address the challenges of modern cloud-native applications, concepts like Platform Engineering are gaining increasing importance. Especially in environments using Azure Kubernetes Services (AKS) , platform engineering plays a crucial role in efficiently managing and scaling containerized applications. What is Platform Engineering and Why is it Important? Platform engineering is the process of designing, implementing, and managing internal platforms that provide developers with a stable and efficient environment. These platforms bundle all the necessary resources and services to ensure smooth development and operation of applications. A well-developed platform engineering team ensures that recurring tasks are automated, allowing developers to focus on writing code without dealing with the underlying infrastructure. In a container environment like AKS, automation and standardization are critical. Platform engineering provides the framework to simplify these complex workflows. How Does Platform Engineering Support AKS Deployments? A key advantage of platform engineering is the ability to standardize the entire lifecycle of applications—from development to testing and deployment. When working with AKS, the main task of the platform engineering team is to create a seamless and scalable environment for container orchestration. Here are some key aspects of how platform engineering supports AKS: Standardizing and Automating Deployments Platform engineering enables the automation of Kubernetes cluster deployments in AKS using best practices and tools such as Infrastructure as Code (IaC) (e.g., Terraform or Azure Resource Manager templates). This automation reduces human errors and accelerates the time needed to deploy applications in production environments. Self-Service Platforms for Developers A well-designed platform engineering team builds self-ser

Multi-Repo to Monorepo: How I Automated 6 Go Microservice Releases and Then Made It 15x Faster Amandeep Singh Amandeep Singh Amandeep Singh Follow Aug 7 Multi-Repo to Monorepo: How I Automated 6 Go Microservice Releases and Then Made It 15x Faster # go # devops # automation # monorepo 6 reactions 1 comment 10 min read