标签：#Uber

Kubernetes makes it easy to define CPU and memory requests for our applications. But there is a problem: How do we know whether those resource requests are actually correct? If an application requests: yaml resources: requests: cpu: "1000m" memory: "1Gi" but normally consumes only a few millicores of CPU and a few megabytes of memory, we may be reserving significantly more cluster capacity than the workload actually needs. I wanted to understand how Kubernetes cost optimization platforms detect this situation, so I built a small hands-on lab using: Google Kubernetes Engine (GKE) CAST AI Kubernetes Docker FastAPI Google Artifact Registry The goal wasn't simply to install CAST AI. I wanted to observe the complete process: Deploy workload ↓ Observe resource usage ↓ Compare requests vs usage ↓ Identify over-provisioning ↓ Generate recommendation ↓ Apply rightsizing ↓ Verify from Kubernetes Architecture The lab architecture was intentionally simple. FastAPI Coffee API | v Docker Image | v Google Artifact Registry | v GKE Cluster | v Kubernetes Deployment | +----------------+ | | v v Pod #1 Pod #2 | | +-------+--------+ | v ClusterIP Service + | v CAST AI | +-------+-------+ | | v v Cost Monitoring Workload Optimization 1. Building a Small Test Application I created a very small FastAPI application for the experiment. from fastapi import FastAPI import socket import os import time app = FastAPI() @app.get("/") def home(): return { "message": "Coffee Shop API", "hostname": socket.gethostname(), "pod": os.getenv("HOSTNAME"), "time": time.time() } @app.get("/coffee") def coffee(): return { "coffee": "Cappuccino", "price": 120 } The hostname in the response was useful later because I could see which Kubernetes Pod handled each request. 2. Containerizing the API The application was packaged using Docker. FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app.py . EXPOSE 8000 CMD ["uvicorn", "app:app", "--host", "0

Building a 100% Local RAG System on Kubernetes — No API Keys Required Ahmed Nafies Ahmed Nafies Ahmed Nafies Follow Jul 30 Building a 100% Local RAG System on Kubernetes — No API Keys Required # kubernetes # rag # llm # postgres 1 reaction Add Comment 8 min read

🇧🇷 Leia a versão em português aqui In Part 5 of this series, we validated the cluster end to end by deploying Nginx. Now let's go one step further: build a custom application's Docker image, publish it, get it running in the cluster, and explore day-to-day operations — version updates, rollback, and scalability (both manual and automatic). As an example, we used a simple REST API ( myapp.war ), built with Spring Boot, purely for illustration — the process applies to any application packaged as a container image. Building the application's Docker image The first step is writing the application's Dockerfile . In this example, a lightweight base image ( alpine ) was used, with Java 11 installed to run the application: FROM alpine WORKDIR /opt/app RUN apk update && apk add vim openjdk11-jre COPY runapp.sh . CMD ash runapp.sh Building the image docker image build -t oregontecnologia/myapp-api:1.0.0 . Publishing the image Before using the image in the cluster, it needs to be available in some registry — either Docker Hub or a private registry . If you'd rather host your own on-premise registry (recommended for corporate environments or those without internet access), check out the companion article on creating a local registry server . To publish to Docker Hub: docker login username: password: docker push oregontecnologia/myapp-api:1.0.0 Deploying the application With the image published, you can check the cluster's current state before proceeding: kubectl get pods -o wide kubectl get deploy -o wide Create the Deployment directly from the command line, pointing to the published image: kubectl create deploy myapp-deploy --image = oregontecnologia/myapp-api:1.0.0 Unlike previous examples in this series (where we used YAML files with kubectl apply -f ), here the Deployment is created directly via the command line with kubectl create deploy . Both approaches are valid — YAML files are more suitable when you need to version and consistently reapply configurations. Exposing the

🇧🇷 Leia a versão em português aqui In previous parts of this series, we built the cluster from scratch: prepared the environment (Part 1), installed containerd and Kubernetes (Part 2), initialized the control-plane (Part 3), and joined the workers (Part 4). With the cluster up and all nodes in Ready state, it's time to actually put it to work: let's deploy our first application. In this article, we'll use Nginx as an example — a classic use case for validating that the cluster is working end to end, from pod creation to service exposure. Organizing the files First, create a directory to organize this deployment's manifests: mkdir nginx cd nginx Keeping Kubernetes manifests organized in per-application directories is a good practice that makes maintenance and versioning (e.g., with Git) easier as the cluster grows. Creating the Deployment A Deployment is the Kubernetes object responsible for managing pod replicas, ensuring the desired number of instances is always running — and handling things like rolling updates and automatic recovery in case of failure. Create the file nginx-deployment.yaml with the following content: apiVersion : apps/v1 kind : Deployment metadata : name : nginx-deployment labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : nginx:1.14.0 ports : - containerPort : 80 This manifest defines: 2 replicas of the Nginx pod ( replicas: 2 ), distributed across the available workers; A selector that ties the Deployment to the pods via the app: nginx label; The nginx:1.14.0 image, exposing container port 80 . Applying the Deployment With the file saved, apply it to the cluster: kubectl apply -f nginx-deployment.yaml kubectl will create the Deployment, and from there Kubernetes takes care of scheduling the 2 pods across the available workers. Checking the Deployment To confirm the Deployment was created and has the desired number of replicas running

🇧🇷 Leia a versão em português aqui In Part 1 of this series, we prepared the environment: defined the hardware, configured /etc/hosts , adjusted the firewall, and disabled SWAP on all nodes. Now that the foundation is ready, it's time to install the container runtime ( containerd ) and the Kubernetes packages themselves ( kubelet and kubeadm ). All the steps below should be run on all servers in the cluster — master and workers — unless stated otherwise. Loading kernel modules Kubernetes, through containerd, depends on two Linux kernel modules: overlay (for the layered filesystem used by containers) and br_netfilter (so that bridge network traffic passes through iptables rules). For these modules to load automatically on every boot, create the file /etc/modules-load.d/containerd.conf : overlay br_netfilter And, to load them immediately (without needing a reboot), run: $ sudo modprobe overlay $ sudo modprobe br_netfilter Adjusting kernel network parameters Create the file /etc/sysctl.d/99-kubernetes-k8s.conf with the following parameters: net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 These parameters ensure that network traffic between pods and services is correctly routed and filtered by Kubernetes. To apply the settings without restarting the server: $ sudo sysctl --system Installing containerd Containerd is the container runtime used by the cluster. In this case, we'll install it through Docker's official repository, using only the containerd.io package (without installing full Docker). 1. Download the repository's GPG key: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg 2. Create the repository file at /etc/apt/sources.list.d/docker.list : deb [ arch = amd64] https://download.docker.com/linux/debian bullseye stable 3. Update the package list and install containerd: sudo apt-get update sudo apt-get install containerd.io 4. Generate the default

🇧🇷 Leia a versão em português aqui This is the first part of a series where I'll share, step by step, how I built my own on-premise Kubernetes cluster, without relying on any cloud provider. The goal is to document the whole process — from environment preparation to a working cluster — as a reference for anyone studying the topic or looking to replicate the same setup at home or at work. I used VPS (Virtual Private Server) and VM (Virtual Machine) for this cluster. However, it can also be set up on physical machines (Bare Metal). Bye the end of this series, it will be easier to understand cloud clusters on AWS (EKS), Google (GKE) and Azure (AKS). In this first part, we'll cover everything needed before installing any Kubernetes component: hardware requirements, basic network configuration, firewall rules, and a few mandatory operating system adjustments. Requirements The following topology was used for this cluster: 3 servers in total 1 master server (control-plane): 2 CPUs (cores) and 2 GB of RAM 2 worker servers (slaves): 1 CPU and 1 GB of RAM each Root access on all machines This is a minimal setup, ideal for study, lab, or testing environments. For production, resources should be scaled according to expected load. Configuring the hosts file Before installing anything, it's important for the machines to resolve each other by name, not just by IP. Edit the /etc/hosts file on all servers and add the corresponding entries: 10 . 0 . 10 . 100 master . company . local master 10 . 0 . 10 . 101 slave01 . company . local slave01 10 . 0 . 10 . 102 slave02 . company . local slave02 This ensures that, later on, the Kubernetes components can correctly resolve node names. Configuring the Firewall Kubernetes depends on specific ports being open between nodes so the control-plane can communicate with the workers (and vice versa). The ports vary depending on the server's role in the cluster. On the master server: Port Protocol 6443 TCP 2379-2380 TCP 10250 TCP 10251 TCP 10252 TCP

Microsoft has released a reference architecture for routing agent traffic on Azure Kubernetes Service. It breaks down the issue into three key choices: which model answers a call, how the call is managed, and which GPU replica handles it. By Claudio Masolo

Foreman is an agentic coder that runs as Kubernetes resources. You describe work as a Workload, it decomposes into tasks, agents running on your nodes pick them up, and a branch comes out the other end with something deterministic standing between that branch and your main. This is the walkthrough. Four objects to understand, an install, an agent, a verifier, and a real run. Every command and every output below is from a working cluster. The four objects Foreman is deliberately small. Almost everything you do is one of these. Agent is a worker definition: which model it talks to, which tools it may call, and what budget it gets. An Agent has a role , and the two that matter here are coder and verifier . Workload is the unit of work you actually author. It carries an intent, a repository, and which agents to use. AgenticTask is what a Workload decomposes into. You rarely write one by hand; you read them to see what is happening. FleetNode is a node that has advertised itself as able to run tasks. The scheduler matches a task's required capabilities against these. The shape of a run is: you apply a Workload, the controller synthesizes AgenticTasks, the scheduler routes each to a FleetNode whose agent can serve it, the agent runs the model in a loop with tools, and the result lands as a branch plus a verdict. The idea underneath it Worth stating plainly, because it shapes every design decision: the model is not trusted, and specifically its claim to have succeeded is not trusted. A coder agent finishes by calling a tool that says "I am done, verdict GO." Foreman treats that as a request, not a result. If the model says GO and produced no diff, the run is recorded as NO-GO. If the verifier's checks do not pass, the work does not land, no matter how confident the summary was. That is the difference between an agent that writes code and a system you can leave running. Everything else in this post is plumbing around that idea. Install Foreman ships as a Helm chart that dep

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry . Project Overview Staxa is a multi-tenant deployment platform I am building solo under Stackforge Labs. The backend is a single Go binary ( staxad ) using the chi router, with about 60 API endpoints, running on K3s on a Hetzner CAX21 ARM64 server that costs around $11/month. Each tenant gets an isolated Kubernetes namespace with their own app container, a PostgreSQL 16 or MySQL 8 database, a subdomain with automatic SSL, and resource quotas. Container builds run through Buildah, and the frontend is Next.js (App Router) with shadcn/ui and Clerk for auth. Bug Fix or Performance Improvement The symptom: POST /api/v1/tenants/{id}/deployments/{depId}/rollback accepted a deployment ID in the URL path and then completely ignored it. Whatever version you asked for, you got the most recent successful deployment instead. The route was wired up correctly in internal/api/router.go:149 : r . Post ( "/tenants/{id}/deployments/{depId}/rollback" , srv . handleRollbackDeployment ) But handleRollbackDeployment never called chi.URLParam(r, "depId") . It read {id} for the tenant and stopped there. How I found it: I was auditing my published API docs against the actual handlers, endpoint by endpoint. When I got to the rollback entry I went to write down what {depId} did, went to the handler to confirm, and found nothing reading it. The docs described an ID that the code never looked at. The worst part is that it returned 202 Accepted and then performed a real, successful rollback. Just not the one you asked for. There was no error to notice, no failed request in any log. The frontend had been passing the deployment ID into the URL since it was written ( src/lib/api.ts ), so the UI always believed the parameter was honored. Root cause: the handler created a rollback deployment row with no reference to any target, and the worker independently decided what to restore. In internal/worker/pipeline.go , runRo

The Go pod is running in production. CPU limit set to 2, metrics look reasonable. But under load, P99 latencies spike intermittently with no obvious cause. No errors, no goroutine leaks, just latency blowing up on traffic bursts. The root cause is usually invisible: GOMAXPROCS equals the number of CPUs on the physical node, not the container limit. Your Go app thinks it has 32 CPUs when it only has 2. The Linux kernel handles the gap in its own way — CFS throttling. What GOMAXPROCS reads (and what it ignores) By default, the Go runtime computes GOMAXPROCS via runtime.NumCPU() , which reads the number of CPUs available at the OS level. On a 32-core Kubernetes node, that returns 32 — regardless of what resources.limits.cpu says in your pod spec. Kubernetes CPU limits are enforced through Linux cgroups (v1 or v2). Cgroups are transparent to processes: a pod with limits.cpu: "2" doesn't see two virtual CPUs, it sees all the node's CPUs and gets suspended when it consumes too much. The Go runtime, historically, never read cgroups. It trusted the physical core count. package main import ( "fmt" "runtime" ) func main () { // Inside a pod with limits.cpu: "2" on a 32-core node fmt . Println ( runtime . NumCPU ()) // → 32 fmt . Println ( runtime . GOMAXPROCS ( 0 )) // → 32 } CFS throttling: how the kernel slows you down The Linux CFS (Completely Fair Scheduler) enforces CPU limits via two cgroup parameters: cpu.cfs_quota_us (allowed CPU time) and cpu.cfs_period_us (measurement window, 100 ms by default). A pod limited to 2 CPUs gets at most 200 ms of CPU time per 100 ms window. When Go spawns 32 OS threads for 32 parallel goroutines, those threads compete for physical CPUs. Once their combined usage exceeds the cgroup quota within the current window, the kernel suspends all threads in the cgroup until the next window starts. That's throttling: a complete application freeze lasting anywhere from a few milliseconds to several tens of milliseconds. A handful of these per second

There's a delightfully literal answer to "Kubernetes with Docker": kind — Kubernetes IN Docker. Each node is a Docker container running a full Kubernetes node image. On an Ubuntu workstation it gives you a real, throwaway, multi-node cluster in about 30 seconds. It's my default for local dev and for CI. Prerequisites on Ubuntu You need Docker Engine and kubectl . If you don't have Docker yet: sudo apt-get update && sudo apt-get install -y docker.io sudo usermod -aG docker $USER && newgrp docker # run docker without sudo Install kind (single static binary): curl -fsSLo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64 chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind kind version A one-command cluster kind create cluster --name dev kubectl cluster-info --context kind-dev docker ps # you'll see a dev-control-plane container — that's your node kind wrote a kubeconfig context for you. Tear the whole thing down just as fast: kind delete cluster --name dev A realistic multi-node cluster Most bugs only show up with more than one node (scheduling, affinity, PodDisruptionBudgets). Define it in a config file: # kind-cluster.yaml kind : Cluster apiVersion : kind.x-k8s.io/v1alpha4 nodes : - role : control-plane kubeadmConfigPatches : - | kind: InitConfiguration nodeRegistration: kubeletExtraArgs: node-labels: "ingress-ready=true" extraPortMappings : - containerPort : 80 hostPort : 8080 protocol : TCP - role : worker - role : worker kind create cluster --name dev --config kind-cluster.yaml kubectl get nodes The extraPortMappings bit is the trick people miss: it forwards a port from your Ubuntu host into the control-plane container, so an ingress controller inside the cluster is reachable at http://localhost:8080 . Loading a locally-built image (no registry needed) This is kind 's best feature for the Docker workflow. Build with Docker, push straight into the cluster's nodes — no registry round-trip: docker build -t myapp:dev . kind load docker-image myapp:dev --name

Welcome back to TechCrunch Mobility, your hub for the future of transportation and now, more than ever, the role AI is playing in it.

If you run GitOps with Flux, a broken manifest usually gets caught the slow way: it merges, the reconciler chokes, and you find out from a failing Kustomization. Flux Schema, the plugin that shipped with Flux 2.9, moves that check left into CI. It validates every YAML document against JSON Schema and CEL rules using the same evaluation logic as the Kubernetes API server, so a bad field fails the pull request instead of the cluster. Install and run it Flux Schema is a CLI plugin, not part of the core binary. Install it through the plugin system: $ flux plugin install schema $ flux schema --help Pin a version in CI so a new release never changes your gate's behavior mid-sprint: $ flux plugin install schema@0.5.0 Point it at a directory of manifests and it validates each document: $ flux schema validate ./manifests It ships with built-in schemas for Kubernetes, OpenShift, Gateway API, and the Flux CRDs, so a fresh install already knows your HelmRelease and Kustomization kinds without any setup. Strict validation flags unknown fields, wrong types, and missing required properties as hard errors, which catches the typos kubectl apply --dry-run=client quietly ignores. What CEL adds over plain schema checks JSON Schema catches shape problems: a string where an int belongs, a misspelled key. CEL rules catch logic problems. Because Flux Schema runs the x-kubernetes-validations rules embedded in CRDs through the same CEL engine the API server uses, a manifest that violates a cross-field constraint (say, a replica count that must stay below a limit, or two mutually exclusive fields both set) fails in CI with the exact message the cluster would have returned. You are testing against the real admission logic, not a stale copy of it. Wire it into a config file Drop a .fluxschema.yml at your repo root to control what gets checked. The file uses the schema.plugin.fluxcd.io/v1beta1 API and a Config kind: apiVersion : schema.plugin.fluxcd.io/v1beta1 kind : Config skipKind : - Secret s

Most of us run kubectl apply -f dozens of times a day without thinking about the machinery it sets in motion. But when something breaks, a Pod stuck in Pending , a Service that won't route, a Deployment that never converges, understanding that machinery is the difference between guessing and debugging. In this article, I'll map the end-to-end flow onto the actual Kubernetes architecture, so you can see not just what happens, but which component is responsible at every step. The Architecture at a Glance Kubernetes is split into two planes: Control plane: the brain. It makes decisions: what should exist, where it should run, and whether reality matches intent. Worker nodes: the muscle. They run your actual workloads and report back. Here's the full picture, with the request flow numbered: (1) apply YAML → API Server (5) Kubelet asks runtime to start container (2) spec persisted in etcd (6) runtime pulls image & runs it (3) controller reconciles spec (7) CNI assigns Pod IP, joins network (4) scheduler assigns a node (8) Kubelet reports status back Now let's walk through the flow, component by component. Step 1: The Cluster Exists Before Your App Does A Kubernetes cluster is the combination of a control plane and a set of worker nodes. The control plane components (API Server, etcd, Controller Manager, Scheduler) can run on dedicated nodes or, in managed offerings like RKE2/EKS/GKE/AKS, be entirely abstracted away from you. Either way, they're always there, always watching. Step 2: You Declare Intent in YAML You don't tell Kubernetes how to run your app, you describe what you want. Typically that's a set of manifests: Deployment: how many replicas, which image, update strategy Service: a stable virtual endpoint in front of ephemeral Pods ConfigMap/Secret: configuration decoupled from the image This declarative model is the foundation of everything that follows. Kubernetes' whole job is to close the gap between your declared state and reality. Step 3: kubectl apply -f Hi

Amazon EKS has recently introduced support for Kubernetes version rollbacks, letting practitioners revert a cluster's control plane to its previous Kubernetes version within 7 days of an upgrade if issues arise. The feature reduces the risk of in-place cluster upgrades by giving teams a safety net to recover quickly from problematic updates. By Renato Losio

minikube is the other "Kubernetes in Docker" option on Ubuntu, and with --driver=docker it runs the cluster inside a Docker container just like kind — but ships with addons (ingress, metrics-server, dashboard, a built-in registry) that make it feel more like a real cluster. Here's a practical setup and how it differs from kind . Install on Ubuntu You need Docker first ( sudo apt-get install -y docker.io , then add yourself to the docker group). Then: curl -fsSLo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 sudo install minikube /usr/local/bin/minikube minikube version Start with the Docker driver minikube start --driver = docker # make it the default so you don't repeat the flag: minikube config set driver docker kubectl get nodes docker ps # a 'minikube' container is your node Size it for real work: minikube start --driver = docker --cpus = 4 --memory = 8g --disk-size = 40g The addons are the reason to pick minikube minikube addons list minikube addons enable ingress minikube addons enable metrics-server minikube dashboard # opens the web UI ingress gives you a working NGINX ingress controller with no manifest wrangling — genuinely useful when you want to test ingress routing locally. The Docker image workflow minikube runs its own Docker daemon inside the node container. The neat trick is pointing your shell's Docker CLI at that daemon, so images you build are immediately visible to the cluster with no push: eval $( minikube docker-env ) # your `docker` now talks to minikube's daemon docker build -t myapp:dev . kubectl create deployment myapp --image = myapp:dev # remember: imagePullPolicy: IfNotPresent so it doesn't try a registry pull Undo it when you're done so docker points back at your host daemon: eval $( minikube docker-env -u ) There's also a built-in registry if you prefer the push model: minikube addons enable registry Accessing services from Ubuntu Two common patterns: # quick tunnel to a single service (prints a

I Run Bare-Metal Kubernetes on $200 of Scrap Hardware (And Why I Burned 3 SD Cards...

The engineering team at Zalando recently described the design and implementation of an in-process, client-side load balancer for a high-throughput API handling around 1 million requests per second. The result was more predictable latency, a drop in infrastructure costs, and better visibility into where failures actually originate. By Renato Losio

The contract between the two companies ends in May 2028, Uber told TechCrunch.

Expedia Group has introduced STAR, an internal AI-assisted observability platform that helps engineers investigate production incidents using service telemetry and LLMs. Built with FastAPI, Datadog, Celery, Redis, and Langfuse, STAR follows structured workflows to analyze telemetry, generate root cause assessments, and support incident response while keeping engineers in the loop. By Leela Kumili