标签：#uber

The Quest Begins (The "Why") I remember the first time I tried to take a weekend side‑project from my laptop to something that felt “real”. I had a cute Express API that talked to Postman, a PostgreSQL container spun up with docker-compose up , and a React front‑end that lived in its own dev server. Everything worked beautifully … until I hit Ctrl+C on my laptop and the whole thing vanished. I needed a way to say, “Hey, keep this running even if I close my laptop, and if something crashes, bring it back up automatically.” I started poking at Docker Swarm, then Nomad, but the docs felt like reading ancient runes. That’s when a coworker slid over a Slack message: “Just try a Kind cluster. It’s K8s locally, and you’ll see why everyone talks about it.” Spoiler: it felt like discovering the secret level in a classic arcade game. Suddenly I could describe what I wanted my system to look like, and the cluster would make it happen — no more babysitting containers. The Revelation (The Insight) Kubernetes isn’t a mystical black box; it’s a declarative orchestrator . You tell it the desired state of your application (how many replicas, which image, what ports to expose) and it works relentlessly to match reality to that state. If a pod dies, Kubernetes spins up a new one. If you ask for three replicas and only two are running, it creates the missing pod. If you update the image tag, it rolls out the change pod‑by‑pod, keeping traffic flowing. Think of it like the save‑game system in a RPG: you define the story you want to experience, and the engine handles the gritty details of loading, saving, and recovering from crashes. The core objects you’ll meet early on are: Pod – the smallest deployable unit (one or more tightly coupled containers). Deployment – manages a set of identical pods, handles updates and rollbacks. Service – a stable network endpoint that load‑balances traffic to a set of pods. Ingress (optional) – exposes HTTP/HTTPS routes from outside the cluster to service

This is Level 5 of our Kubernetes networking series. So far, we've built up a solid foundation: LEVEL 1 — Pod networking LEVEL 2 — Pod-to-Pod communication LEVEL 3 — Service (a stable internal endpoint) LEVEL 4 — DNS (service name → Service IP) But we still have a glaring gap: how does a real user on the internet actually reach your Kubernetes application? That's exactly what this article covers — Ingress, Ingress Controllers, and the newer Gateway API. Table of Contents The Problem: The Internet Can't Reach a ClusterIP The Basic Solution: Ingress and Gateway API What Is Ingress? A Routing Example Ingress Is Not the Actual Proxy A Simple Analogy: Traffic Police A Basic Ingress YAML Example Breaking Down the Key Fields Host-Based Routing Path-Based Routing Why Not Just Use a LoadBalancer Service for Everything? The Complete Traffic Flow Where Does DNS Fit In? The Ingress Controller A Typical Architecture Ingress vs Service Ingress vs LoadBalancer Service HTTPS and TLS Termination Why Terminate TLS at the Edge? Referencing a TLS Certificate Routing Multiple Domains The Gateway API GatewayClass, Gateway, and HTTPRoute Ingress vs Gateway API Important Distinctions: Ingress Is Not CNI or Service Troubleshooting Ingress Layer by Layer Common Ingress Mistakes The Complete Kubernetes Networking Picture (Levels 1–5) The Mental Model to Memorize Level 5 Checkpoint What's Next: NetworkPolicy The Problem: The Internet Can't Reach a ClusterIP Suppose you want users to reach your application at myapp.example.com . Inside your cluster, you have: Service : frontend ClusterIP : 10.96.20.10 frontend Service ├── Pod 1 ├── Pod 2 └── Pod 3 A user on the internet can't simply visit http://10.96.20.10 — that's a private Kubernetes Service IP, invisible outside the cluster. We need something sitting at the edge of the cluster to bridge that gap. The Basic Solution: Ingress and Gateway API Historically, Kubernetes solved this with Ingress . More recently, Kubernetes introduced a more expres

How we turned always-on Kafka sinks into on-demand workers that shrug off nightly bombardments — by scaling on the right signal, tuning per-pod drain rate, and keeping autoscaling from sabotaging itself. Every number in this post is measured from a local lab you can run yourself — the full code is on GitHub , and the Appendix has the commands. The problem We run a fleet of Kafka sinks — consumer services that read change events from Kafka, apply business logic, and write the result into a service-local database as a query-friendly materialized view. It keeps reads fast and independent from upstream systems, and it's a great pattern. But the workload has an awkward shape. Most sinks are idle most of the day, then buried in minutes. Traffic isn't steady: changes arrive in bursts, usually from nightly imports or CDC jobs. The rest of the day the topic is quiet. topic activity over 24h msgs ▲ │ ██ nightly import / CDC burst │ ██ │______________██______________ flat, idle ~22h/day └───────────────────────────────▶ time That shape creates two problems at once : Idle waste. When the topic is quiet, each sink still runs — it polls Kafka, holds connections, emits metrics, and occupies CPU and memory. Multiply one "small" sink across dozens of them and several regions, and you're paying around the clock for work that happens for a couple of hours a night. Spike lag. When the burst lands, a backlog builds fast. If consumers can't drain it quickly enough, consumer lag — the gap between what's been produced and what's been processed — climbs, and downstream reads start serving stale data. We want two things that sound contradictory: cost almost nothing when idle , and absorb the spike fast when it hits. Why the obvious autoscaler doesn't help The reflex is a Kubernetes Horizontal Pod Autoscaler (HPA) on CPU or memory. For sinks, that's the wrong signal. Sink work is I/O-bound : the consumer spends its time waiting on Kafka polls and database writes, not burning CPU. So when a ba

Most Spring Boot services end up with a scheduled job in them somewhere. A nightly reconciliation, a report, an export to some partner system. It starts small, and it goes in the main app because that's where the domain code already is. One artifact, one deployment, one pipeline. That's a real advantage and it's why most teams do it. This post is about when that stops being a good trade, how to split the job out, and when you shouldn't. The memory problem Look at how much memory each workload uses over a day. The API is fairly flat. Warm heap, connection pool, some caches. It moves with traffic but it doesn't swing much. The batch job uses close to nothing for 23 hours, jumps while it runs, then drops back to nothing. When both live in the same JVM, the pod has to be sized for the peak. So every replica of your API holds batch-sized memory all day, for a job that runs once. With three replicas you're reserving that headroom three times over so one job can use it once, at 2am. Memory limits are not like CPU limits CPU is compressible. Go over your CPU limit and the kernel throttles you. The app gets slower and keeps running. Memory doesn't work that way. There's no "run with less" mode. If the container goes over its memory limit, the kernel kills the process. What you get is a container that exited with code 137 (that's 128 + 9, where 9 is SIGKILL). What you don't get is anything useful in the logs. No OutOfMemoryError , no stack trace, no heap dump unless you configured one and it had time to write, no shutdown hook. The JVM was running fine, asked for another page of memory, and got killed for it. So a batch job sharing a pod with your API is a way for a nightly job to take down the pods serving traffic. If the job's working set grows (bigger dataset, a table that keeps growing, one unusually heavy day) the thing that dies is the API. There's a quieter version of the same problem. Even when the job stays under the limit, it allocates heavily and triggers longer GC

The partnership is expanding beyond the initial market of Zagreb, Croatia to four additional European cities.

The Kubeflow project has unveiled several technical updates to enhance distributed AI and high-performance computing on Kubernetes. These advancements include Kale 2.0, a modernised SDK with native Spark support, and expanded capabilities for the Kubeflow Trainer. The developments arrive as the project moves towards graduation from the Cloud Native Computing Foundation. By Matt Saunders

Scheduled LLM curation is the least glamorous agent workload you run. A cron job wakes up at 3am, reads a pile of memory, asks a model to dedupe it, summarize it, re-rank it, and writes the result back. Nobody is watching. There's no chat window, no streaming tokens, no human to click a button. It just has to work, quietly, every night. That "nobody is watching" part is exactly what makes the cloud-versus-local decision harder than it looks. When you have a human in the loop, a failed API call throws an error you can see and retry. In a headless cron context, the same failure turns into a job that hangs on an approval prompt no one will ever answer, or a pod that curated three months of context into an emptyDir that vanished on restart. I've run curation both ways: nightly jobs hitting a hosted API, and the same logic pointed at a local model on my Kubernetes cluster. Both work. They fail differently, cost differently, and demand different things from you operationally. Here's the actual tradeoff, not the marketing version. The decision point You reach this fork once your agent memory stops being a toy. Early on, you curate by hand or with a cheap synchronous call inside your agent loop. Then the memory grows, the curation gets expensive, and you pull it out into a scheduled job so it runs off the critical path. Now you're paying an API on a timer, and two things start to bug you. First, the data. Curation reads your entire memory store to make decisions. If that memory contains anything you'd rather not stream to a third party (internal notes, customer context, infrastructure details), every scheduled run ships it over the wire. I wrote about the general version of this problem in privacy-routed LLM inference , and scheduled curation is the workload where it bites hardest, because it touches everything, repeatedly, forever. Second, the cost shape. A curation pass over a large vector store is a lot of tokens for a job that produces no user-facing latency benefit. Yo

Most Kubernetes advice assumes you have a platform team: specialists who own upgrades, ingress, security policies, and the 2 a.m. pages. The teams I am writing for usually have three to ten engineers, one of whom “knows Kubernetes,” and no dedicated platform team. They depend on a cluster that nobody fully owns. I work in enterprise environments where platform teams are large and everything is process. This article is the opposite exercise: what is the minimum discipline a small team needs to run Kubernetes in production—and what enterprise baggage should it refuse to copy? The question that matters more than any tool Before any checklist: who owns the platform after the migration is finished? Not “who set it up.” Who owns upgrades next year, certificate renewals, the CNI version, and deprecated APIs? If the answer is one person's name, you do not have a platform. You have key-person risk with YAML on top. If the answer is “nobody, really,” Kubernetes is invisible operational debt accumulating interest. The rest of this checklist exists to make that ownership small enough for a small team to carry. For each item, score 0 if it does not exist, 1 if it exists but is informal or untested, and 2 if it is documented and tested. The purpose is not to produce a flattering number. It is to expose the next few conversations the team needs to have. 1. Deployments: Git is the source of truth Treat Git as the source of truth for workloads and cluster configuration, including temporary fixes. Use one reconciliation path—for example, Argo CD or Flux—so production changes are reviewed and reproducible. Keep emergency access, but reconcile every emergency change back into Git. Define and test a rollback path for every service. A Git revert is useful only if your delivery process can deploy it safely. This converts your cluster from a mystery into a diff. Every other practice gets easier once “what is running?” has an answer. 2. The rollout basics that prevent late-night incidents R

KEDA 3.0 just landed, and the headline feature is the one I care about most as someone who watches a cloud bill: event-driven autoscaling now covers 80+ event sources (Kafka, RabbitMQ, and a long list more) with proper scale-to-zero. If you run workloads that sit idle most of the day and spike when work arrives, this is the difference between paying for capacity you use and paying for capacity that waits. I have been moving our intermittent workloads onto this pattern, so here is what scale-to-zero actually does to the bill, where it helps, and the sharp edges nobody mentions. The problem: HPA scales to one, not to zero Standard Horizontal Pod Autoscaler has a floor. minReplicas cannot be zero, so a workload that processes a queue twice a day still keeps at least one pod (and often the node under it) running 24/7. For a consumer that is busy 2 hours a day, you are paying for 22 hours of nothing. KEDA changes the shape of the question. Instead of "how many replicas does current CPU justify," it asks "are there events waiting." No events, zero pods. Events arrive, it scales from zero up to whatever the load needs. That floor of zero is the whole game for intermittent work. Where scale-to-zero actually pays off Not every workload benefits. The ones that do share a profile: bursty, event-triggered, and tolerant of a short cold start. In our environment the clear wins were: Queue consumers. A worker draining an SQS or RabbitMQ queue that fills a few times a day. Idle 80%+ of the time, now scales to zero between bursts. Kafka stream processors for low-volume topics that only see traffic during business hours. Scheduled batch jobs dressed up as long-running services because nobody wanted to re-architect them. Scale-to-zero gets most of the savings without the rewrite. Dev and staging consumers that had no reason to run overnight and did anyway. A rough sizing rule I use: if a workload is idle more than half the day and an extra few seconds of latency on the first event is

An extortion gang known for targeting transportation companies and private equity firms has taken credit for a breach at Uber Freight.

Netflix migrated most of its batch workloads onto Kueue, an open-source cloud-native batch job execution system that has outgrown its homegrown solution over the years. The company mapped the capabilities previously created in-house to Kueue’s functionality and also benefited from new features that would have been costly to incorporate into its homegrown solution. By Rafał Gancarz

Docker and Kubernetes are two of the most consequential infrastructure technologies of the last decade. They changed how software is built, packaged, and deployed. They are also two technologies that most engineers use before they understand, which creates gaps in knowledge that show up at the worst times: a production outage, a security incident, a performance problem you cannot diagnose. This guide builds understanding from the ground up. Every concept is introduced with the problem it solves. You will understand why containers exist before you understand what they are. You will understand why Kubernetes exists before you understand how it works. By the end, you will know not just how to run these technologies but how to reason about them. Table of Contents The Problem Containers Solve - Why Docker Exists Docker Internals - What a Container Actually Is Images - Building Portable Application Packages Dockerfile - Writing Reproducible Builds Docker Networking - Container Communication Docker Volumes - Managing State Docker Compose - Multi-Container Applications The Problem Kubernetes Solves - Why Orchestration Exists Kubernetes Architecture - The Control Plane and Data Plane Core Kubernetes Objects - Pods, Deployments, Services, ConfigMaps, Secrets Namespaces and RBAC - Multi-Tenancy and Access Control Storage in Kubernetes - Persistent Volumes Ingress - Routing External Traffic Helm - Package Management for Kubernetes Service Mesh - Istio and Advanced Traffic Management AWS Container Services - ECS and EKS Real Architecture Patterns The Problem Containers Solve - Why Docker Exists The Classic Failure Mode A developer builds an application on their MacBook. It works. They hand it to the QA team. It does not work. They hand it to the operations team to deploy to production. It works differently than in QA. "It works on my machine" is not a joke. It is a description of a real, chronic infrastructure problem. The application depends on: A specific version of Python, No

n today's world of cloud-native development, businesses require powerful, scalable, and flexible platforms that help developers efficiently build and operate their applications. An Internal Developer Platform (IDP) based on Azure Kubernetes Services (AKS) provides an optimized environment that brings together all the key components for modern software engineering. This article explains how to develop such a platform using AKS, what key components are required, and how to integrate them optimally. What is an Internal Developer Platform (IDP)? An internal developer platform is a set of tools, processes, and automations provided to developers to simplify the entire software development process. It offers a standardized environment where developers can write, test, and deploy code without worrying about the infrastructure or underlying complexities. An IDP built on Azure Kubernetes Services (AKS) also allows for the operation of containerized applications in a fully managed, highly available, and scalable environment. Core Components of a Development Platform on AKS When building an internal developer platform based on AKS, several key components ensure an efficient and robust system. Here are the essential elements: Azure Kubernetes Services (AKS) as the Central Platform AKS forms the core of the development platform. It provides a scalable and managed Kubernetes environment where all containerized applications run. With full integration into other Azure services, developers can access a wide range of tools to efficiently manage, monitor, and scale their workloads. Service Mesh for Managing Microservices Communication In a microservices architecture, which is commonly used in modern cloud-native applications, communication between services plays a crucial role. A Service Mesh like Istio or Linkerd enables the management and monitoring of this communication. It provides features such as load balancing, traffic management, security policies, and monitoring for microservi

I Automated My Entire GitOps Security Stack. The First Thing It Blocked Was My Own...

The divestiture comes as the two once-tight companies have started to diverge on the business side.

The Nautilus DevOps team is gearing up to deploy applications on a Kubernetes cluster for migration purposes. A team member has been tasked with creating a ReplicaSet outlined below: Create a ReplicaSet using nginx image with latest tag (ensure to specify as nginx:latest ) and name it nginx-replicaset . Apply labels: app as nginx_app , type as front-end . Name the container nginx-container . Ensure the replica count is 4 . Solution Step 1: Generate the ReplicaSet YAML First, let's generate a base ReplicaSet manifest: kubectl create replicaset nginx-replicaset \ --image = nginx:latest \ --dry-run = client -o yaml > nginx-replicaset.yaml Step 2: Edit the YAML to Add Requirements Open the file and modify it according to the requirements: nano nginx-replicaset.yaml Update the file with: Replica count: 4 Labels: app: nginx_app , type: front-end Container name: nginx-container Here's the complete YAML: apiVersion : apps/v1 kind : ReplicaSet metadata : name : nginx-replicaset labels : app : nginx_app type : front-end spec : replicas : 4 selector : matchLabels : app : nginx_app type : front-end template : metadata : labels : app : nginx_app type : front-end spec : containers : - name : nginx-container image : nginx:latest ports : - containerPort : 80 Step 3: Apply the ReplicaSet Create the ReplicaSet in your cluster: kubectl apply -f nginx-replicaset.yaml Expected output: replicaset.apps/nginx-replicaset created Step 4: Verify the ReplicaSet Check that the ReplicaSet was created successfully: kubectl get replicasets Expected output: NAME DESIRED CURRENT READY AGE nginx-replicaset 4 4 4 10s Step 5: Verify Pods Check that 4 pods were created: kubectl get pods Expected output: NAME READY STATUS RESTARTS AGE nginx-replicaset-xxxxx 1/1 Running 0 15s nginx-replicaset-yyyyy 1/1 Running 0 15s nginx-replicaset-zzzzz 1/1 Running 0 15s nginx-replicaset-wwwww 1/1 Running 0 15s Step 6: Verify Labels Check that the labels are correctly applied: kubectl get replicaset nginx-replicaset --s

With the stateless MCP spec now officially out as of July 28th, 2026, there are now two methods of connecting to and configuring an MCP Server. In this blog post, you'll learn what the stateless MCP spec means for the future, the breakdown of the spec, and how to implement it. 💡I wrote a "engineering details quickstart" for some of the other changes that came with the new spec as well, which you can find here. Stateless MCP Breakdown Two of the key changes in the 2026-07-28 change: removal of initialization handshakes and session IDs. An initialization handshake was the startup exchange used by MCP versions through 2025-11-25. Client sends an initialize request containing its protocol version, capabilities, and client information. Server returns an InitializeResult with the negotiated version, server capabilities, and server information. It could also return MCP-Session-Id Client sends notifications/initialized. Normal MCP requests begin. It established what features both sides supported before tools or resources were used. In MCP 2026-07-28 , this handshake was removed. Each request instead carries its protocol version and client metadata, making requests independently processable and stateless. Session IDs were also removed, as anything with a session ID is stateful, since that ID serves as a lookup key for data stored on a server or in a database. Example: If you log into Gmail and look at the devices that are logged into Gmail (your phone, laptop, etc.), the reason you don't need to continuously log into them/daily login is that a session exists for that device. Headers and Body Some headers must be in the body, and some headers that aren't. Standard HTTP headers (Content-Type, accept, Content-Length, etc.) don't need to be in the body. MCP headers that mirror requests, however, need to be in both the header and the body. MCP-Protocol-Version Mcp-Method Mcp-Name Mcp-Param-* Notice how in the example below you'll see the name, method, and protocol version are in

Digital transformation has led companies to organize their infrastructure and development processes in entirely new ways. To address the challenges of modern cloud-native applications, concepts like Platform Engineering are gaining increasing importance. Especially in environments using Azure Kubernetes Services (AKS) , platform engineering plays a crucial role in efficiently managing and scaling containerized applications. What is Platform Engineering and Why is it Important? Platform engineering is the process of designing, implementing, and managing internal platforms that provide developers with a stable and efficient environment. These platforms bundle all the necessary resources and services to ensure smooth development and operation of applications. A well-developed platform engineering team ensures that recurring tasks are automated, allowing developers to focus on writing code without dealing with the underlying infrastructure. In a container environment like AKS, automation and standardization are critical. Platform engineering provides the framework to simplify these complex workflows. How Does Platform Engineering Support AKS Deployments? A key advantage of platform engineering is the ability to standardize the entire lifecycle of applications—from development to testing and deployment. When working with AKS, the main task of the platform engineering team is to create a seamless and scalable environment for container orchestration. Here are some key aspects of how platform engineering supports AKS: Standardizing and Automating Deployments Platform engineering enables the automation of Kubernetes cluster deployments in AKS using best practices and tools such as Infrastructure as Code (IaC) (e.g., Terraform or Azure Resource Manager templates). This automation reduces human errors and accelerates the time needed to deploy applications in production environments. Self-Service Platforms for Developers A well-designed platform engineering team builds self-ser

AWS launched Dogwood this week — an open-source policy language (Apache 2.0) for AI agent runtime verification. It extends Cedar, AWS's existing authorization language (now a CNCF sandbox project), with something Cedar fundamentally can't do: reason about sequences of actions over time. "Point-in-time decisions make sense for many forms of access control, but when agents compose multiple actions into longer workflows, the sequence itself becomes something teams want to govern." That's the gap Dogwood fills. What Cedar couldn't do Cedar is stateless. You give it a request — principal, action, resource, parameters — and it returns allow or deny. Given the same request, Cedar always returns the same answer, regardless of what happened five minutes ago. That's a useful property for analysis, but it's a blind spot for agents. Consider: an agent is restricted to transferring no more than $5,000 per hour. If Cedar only evaluates the current request against completed transfers, the agent can fire off three concurrent $2,000 requests before any of them finish. Each looks fine in isolation. The total blows the limit. Dogwood has the event history. It counts all transfer requests — including those currently in-flight — so the third $2,000 request gets denied even before the first two complete. What Dogwood adds Dogwood introduces temporal conditions that examine earlier tool calls and their results. You can: Check whether an event occurred — e.g., was approval granted for this exact stock/quantity in the last hour? Count calls in a time window — rate limiting across concurrent requests Count distinct values — e.g., how many unique payment recipients this session Sum values — total transferred, total refunded The stock trading example from AWS is the clearest illustration: an agent may only sell shares if an approval tool returned a positive response for that stock and share count within the previous hour. That approval is a separate event the policy engine finds in the agent's

If you've run Kubernetes for more than a day, you've seen this: apiVersion : v1 kind : Secret metadata : name : db-credentials type : Opaque data : username : YWRtaW4= password : c3VwZXJzZWNyZXQ= And somewhere in the back of your mind you filed it under "encrypted credentials." It isn't. Those values are Base64, and Base64 is encoding, not encryption. YWRtaW4= is just admin written in a different alphabet — reversible instantly, by anyone, with no key. This trips up an astonishing number of teams, so let's clear it up for good. Prove it in one command kubectl get secret db-credentials -o jsonpath = '{.data.password}' | base64 --decode # supersecret No key. No password. No "decryption." Base64 is a binary-to-text encoding — its entire job is to represent arbitrary bytes using a safe 64-character alphabet so they survive transport and storage in text-based systems (etcd, YAML, JSON, HTTP headers). Kubernetes encodes Secret data values purely so binary values (certs, keys, gzip blobs) can live inside a YAML/JSON object. That's it. Security was never the point. If you want to eyeball a whole Secret at once instead of decoding fields one by one, I built a small in-browser tool for exactly this — paste the YAML and it decodes every data: value locally (nothing is uploaded): Kubernetes Secret Decoder . (Disclosure: it's my free, no-ads tool.) data vs stringData A quick related gotcha: data expects Base64 , but stringData expects plain text and Kubernetes Base64-encodes it for you on write: stringData : password : supersecret # plain text; k8s encodes it into data.password Both end up identically un-secret at rest. So what actually protects a Secret? Base64 gets you nothing here. Real protection is layered: Encryption at rest for etcd — configure a KMS provider (AWS/GCP/Azure KMS) or at minimum aescbc / secretbox via an EncryptionConfiguration . Without this, Secrets sit in etcd Base64-only. Sealed Secrets (Bitnami) — encrypt secrets before they hit Git; only the in-cluster