标签：#Uber

Originally published at woitzik.dev Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily. Every internal service in my homelab goes through the same authentication gate: Authelia. Proxmox, PBS, Grafana, ArgoCD, Headscale, ArgoCD, Uptime Kuma, Paperless, Nextcloud — 25+ web services, one login, one session, one set of access rules. The OIDC provider, the Postgres backend, the session store, and the secrets are all running inside k3s, backed by CNPG, Redis, and Vault. This article is the full implementation: how the pieces fit together, why certain design decisions were made, and the specific bugs that bit me along the way. View the complete homelab infrastructure source on GitHub 🐙 The Architecture Authelia runs as a Kubernetes Deployment in the apps namespace, protected by the same default-deny NetworkPolicy that applies to everything else. It has three dependencies: PostgreSQL — CNPG-managed postgres-authelia cluster in the database namespace Redis — session store, ephemeral (no persistence needed) Vault — hmac_secret, OIDC private keys, JWT secrets, session secrets The Traefik ForwardAuth middleware sits in front of every service. When a request hits Traefik, the middleware sends a verification request to Authelia's /api/verify endpoint. Authelia checks the session cookie, validates the OIDC token if applicable, and returns a 200 (allowed) or 401 (redirect to login). # kubernetes/apps/authelia/middleware.yml apiVersion : traefik.io/v1alpha1 kind : Middleware metadata : name : authelia namespace : apps spec : forwardAuth : address : " http://authelia.apps.svc.cluster.local:9999/api/verify" trustForwardHeader : true authResponseHeaders : - Remote-User - Remote-Groups - Remote-Email Every IngressRoute that needs protection adds middlewares: [{name: authelia}] . Services that need API-level protection (not browser-based) use OIDC

The Problem with Fully Autonomous Remediation Every platform team eventually asks the same question: can we let something automatically fix production when it breaks? The instinct to say yes is understandable incidents at 3 a.m. are expensive, and a lot of Kubernetes failures follow recognizable patterns. But fully autonomous remediation has a bad failure mode: when the agent is wrong, it's wrong fast, and it's wrong at scale. AIOps agents for Kubernetes solve this by splitting the problem in two: let the agent do the work of detection, correlation, and proposal the parts humans are slow and inconsistent at and keep a human as the final decision-maker for anything with real consequences. This is the human-in-the-loop (HITL) model, and on Google Cloud it maps cleanly onto existing primitives: GKE for the runtime, Cloud Monitoring/Logging for signal, IAM and Kubernetes RBAC for guardrails, and Vertex AI or a self-hosted model for the reasoning layer. What the Agent Actually Does Strip away the buzzwords and an AIOps agent for Kubernetes does four things on a loop: Watch — consume events, metrics, and logs from the cluster and surrounding GCP services Correlate — connect a symptom (say, elevated 5xx rate) to a likely cause (a bad rollout, a starved node, an expired credential) Propose — generate one or more candidate remediations, each with a confidence score and an estimate of blast radius Act or Ask — execute directly if the action is pre-approved as low-risk, otherwise route to a human for a decision The engineering effort is disproportionately in steps 2 and 4. Step 2 (correlation) requires the agent to reason over multiple, often noisy signal sources rather than pattern-match a single metric. Step 4 (the human gate) requires a review surface good enough that a tired on-call engineer can make a correct decision in seconds, not minutes. Core Signals on GKE The Approval Gate, Concretely The human-in-the-loop gate is usually a chat-based approval flow, since on-call e

In today's digital world, businesses face the challenge of developing, deploying, and scaling applications faster and more efficiently. One of the key technologies supporting this transformation is container technology. What are Containers and Why Are They Important? Containers allow applications to be packaged into lightweight, self-contained, and portable units that can run consistently in any environment—from a local development machine to a cloud platform. This reduces dependencies and significantly simplifies application deployment and scalability. Unlike virtual machines (VMs), containers share the operating system kernel, making them more resource-efficient. This leads to higher efficiency and allows businesses to run more applications on the same infrastructure. Introduction to Kubernetes: Orchestration of Containers While containers represent a revolutionary approach to developing and running applications, it’s not enough to simply have containers. Once applications consist of dozens or hundreds of containers, managing, orchestrating, and scaling them becomes critical. This is where Kubernetes comes in. Kubernetes is the world’s most widely used container orchestration platform. It enables the automatic deployment, scaling, and management of containerized applications in clusters. With Kubernetes, companies can ensure their applications are always available, automatically recover from failures, and roll out new versions without downtime. Azure Kubernetes Services (AKS): Kubernetes in the Cloud Azure Kubernetes Services (AKS) is Microsoft’s fully managed Kubernetes solution. With AKS, businesses benefit from simplified Kubernetes deployment by offloading infrastructure management to Microsoft. This means you can focus on developing and scaling your applications while AKS simplifies the management and maintenance of Kubernetes clusters. Benefits of AKS: Fully managed: AKS takes care of the management and patching of Kubernetes, allowing businesses to focus on

Part 3 of the User Connectivity Architecture series. Introduction The first post in this series described the pattern: a heartbeat on a timer, an Event Hub, a worker writing sessions into Redis, and Redis key expiration driving facility online/offline status. One detail matters later. The heartbeat interval is not hard-coded in the client. The API tells the client when to call next, and the default is 30 seconds. The second post covered two years of running that in production. This post is about the month it stopped working. In January 2026 our heartbeat traffic went from boring to terrifying and stayed there for about four weeks. This is the story of what broke, why the original design had a ceiling we never noticed, and the changes that fixed it: more Event Hub partitions, Azure Container Apps, and KEDA . The Storm A normal day looked like this: 51,000-58,000 heartbeats per hour , hour after hour Roughly 15-16 events per second at idle Flat, predictable, forgettable On January 5, around 7:00 AM PST , it stopped being flat. Time (PST) Heartbeats/hour Baseline ~57,000 12:00 PM 80,005 1:00 PM 216,351 5:00 PM 343,480 9:00 PM 466,760 That is eight times normal event volume in a single hour, and it was still climbing. Events were only half the story. SignalR connection counts told the other half. At the worst of it we were holding roughly eleven times the connections we normally maintain, and every one of those was a browser session we had to track, keep alive, and report status for. It did not spike and recover. It stayed elevated for weeks while we hunted for the cause. When we finally found it, the answer was almost funny: 507 zombie sessions that never ended, running months-old cached client code, and a single user account responsible for 33% of all our token API traffic . One account. Eight times the load. Four weeks. What Eight Times Load Actually Did Here is the part that matters, and it has nothing to do with the number itself. Our Event Hub had one partition. I

Having a platform is not enough; the real challenge is ensuring that it is understandable, usable, and actually adopted by its users. A capability is done when it can be reliably used by others. To evaluate progress, you can ask yourself “Is this being used?” and “Does it reduce friction for users?” This can help align development work with actual user value rather than delivery, By Ben Linders

I Replaced kube-proxy with eBPF in Production (And Why My Monitoring Went Blind for 6...

Running AI agents on Kubernetes raises a key question: should each agent get its own Pod? The kagent project argues no—agents are bursty, short-lived, can spawn subagents, and may wait for human approval, making one Pod per agent wasteful. Agent-substrate adds a control plane to schedule logical “Actors” onto long-lived worker Pods. By Mark Silvester

Kalanick continues to get the band back together, after acquiring Anthony Levandowski's autonomy startup, and even soliciting investment from Uber itself.

After Uber and Waymo ended their partnership in Phoenix earlier this year, experts and robotaxi watchers wondered whether the companies' improbable bromance was fraying. Not so, Uber CEO Dara Khosrowshahi said today. The two companies are committed to continue working together in Atlanta and Austin, and the partnership remains "very strong." "Waymo is a very […]

Series: Platform engineering with Go | Topics: Go, Kubernetes, Cobra, client-go, metrics-server, Platform Engineering This is part of the Platform Engineering with Go series. This post builds on the Cobra CLI patterns from post 4 and client-go from post 3. Read post 4 first if you haven't yet. kubectl top tells you what's happening. It doesn't tell you how close to the edge you are. In post 3 and post 4 , we built a health reporter and learned how to structure a Go CLI with Cobra. Now we put both together into something with real operational value. kubectl top pods -n production NAME CPU ( cores ) MEMORY ( bytes ) go-api-7d6b9f8c4-xk2pq 240m 490Mi go-api-7d6b9f8c4-mn9rt 180m 210Mi go-api-7d6b9f8c4-p8wvz 200m 198Mi That first pod is using 490Mi of memory. Is that fine or is that a problem? Without knowing the limit, you can't tell. You'd have to run kubectl describe pod go-api-7d6b9f8c4-xk2pq , find the resources section, do the mental arithmetic, and repeat for every pod you care about. ferctl top does all of that in one command: ferctl top -n production NAMESPACE NAME CPU USE CPU REQ CPU LIM CPU% MEM USE MEM REQ MEM LIM MEM% STATUS production go-api-7d6b9f8c4-xk2pq 240m 250m 500m 48% 490Mi 256Mi 512Mi 95% !! CRITICAL production go-api-7d6b9f8c4-mn9rt 180m 250m 500m 36% 210Mi 256Mi 512Mi 41% OK production go-api-7d6b9f8c4-p8wvz 200m 250m 500m 40% 198Mi 256Mi 512Mi 38% OK One pod is at 95% of its memory limit. In production, that's a page waiting to happen. ferctl top catches it before it becomes an incident. What you'll learn How to extend the Cobra CLI structure from post 4 with a real subcommand How to query the metrics-server API using k8s.io/metrics How to correlate live metrics with pod specs to show usage vs limits How to implement configurable near-limit warnings How to format clean aligned output with tabwriter How to verify the tool against your real minikube cluster Prerequisites Posts 1–4 read; client-go patterns from post 3 , Cobra CLI structure from pos

HashiCorp has released a public beta of Vault Kubernetes key management, a KMS v2-compatible plugin that lets the Kubernetes API server delegate envelope encryption to Vault Enterprise, moving the key encryption keys that protect etcd data out of the cluster and into a separately governed trust domain. By Mark Silvester

Building an AI agent locally is an exciting first step. Running that same agent reliably in production is a different challenge. Once real users and external services are involved, the application needs more than working code. It needs repeatable deployments, secure configuration, health checks, monitoring, controlled updates, and a clear recovery process. This article is part of my MCP series. If you are new to the topic, start with my first article: Model Context Protocol (MCP) Servers Explained: A Complete Beginner’s Guide . In this article, I will outline a practical architecture for taking a Model Context Protocol, or MCP-based, AI agent from a local development environment to Kubernetes. This is a production architecture blueprint. The exact implementation will depend on the AI provider, MCP servers, cloud platform, and security requirements used by the application. What Is an MCP-Based AI Agent? The Model Context Protocol provides a standardized way for AI applications to connect with external tools, services, and data sources. An MCP-based agent may interact with: Internal APIs Databases File systems Search services Monitoring platforms Business applications Custom automation tools A basic implementation might work well on a developer's machine. In production, however, every dependency introduces operational questions: How will the application be deployed? Where will credentials be stored? How will failed requests be detected? Can the service handle additional traffic? How can a broken release be rolled back? What happens when an MCP server becomes unavailable? These are familiar DevOps and Site Reliability Engineering problems applied to a new type of workload. Target Architecture A practical delivery flow could look like this: Developer ↓ GitHub Repository ↓ GitHub Actions ↓ Container Registry ↓ Kubernetes Cluster ↓ MCP Servers and External Services ↓ Logs, Metrics, Traces, and Alerts Each component has a clear responsibility: GitHub stores the application

Every year, developers ask the same question: "Which programming language should I learn next?" And...

Originally published at wostal.eu . TL;DR : My Hetzner k3s lab quietly became a platform. Dozens of operators with leader-election leases hammered the default datastore — SQLite via kine — until compaction entered a death-spiral: 1.36M rows, a 13.8 GB WAL that wouldn't checkpoint, CPU pinned at 99%, load average 79 on 8 cores. I stopped the bleeding by truncating the WAL, then migrated the control plane to embedded etcd (7.5 GB SQLite → 313 MB etcd, load 79 → 5). This is the full postmortem — and the lessons. This is a war story, not a tutorial. It's about the moment a homelab stops being a homelab and starts behaving like production — without ever announcing it. The cluster in question, homelab , is the Hetzner k3s setup I wrote about previously . It started small. It did not stay small. In this post I'll cover: How an overgrown lab broke the default datastore — the kine/SQLite compaction death-spiral The firefight — measuring instead of guessing, and the fix that actually worked The permanent fix — migrating the control plane to embedded etcd, and the honest caveats The meta-lesson — how to recognize when your lab has become a platform A diagnostic runbook — so next time it's minutes, not hours There's a companion piece to this incident. The CI pipeline that ran this etcd migration was itself freshly — and badly — migrated, and debugging it cost me hours over a single missing newline. I split that into its own post: I Let an AI Re-Platform My CI Pipeline. Here's What Broke. Context: it's "just a homelab" — except it isn't homelab began like any homelab: one k3s node on Hetzner, a few things to play with. The problem is that over months it quietly became a platform . A single master node ( cx43 , 8 vCPU / 16 GB, untainted, and also carrying Longhorn and workloads) now runs: ArgoCD, Kargo, Crossplane/Upbound, CloudNativePG, EMQX, Longhorn, trivy-operator, kubescape, Gatekeeper, Goldilocks/VPA, VictoriaMetrics, Loki, OpenTelemetry, Argo Workflows/Events/Rollouts, kga

Welcome back to TechCrunch Mobility, your hub for the future of transportation and now, more than ever, the role AI is playing in it.

Uber has partnered with — and in some cases made direct investments in — about 30 autonomous vehicle companies over the past two years. Here's the list and the latest on the partnerships.

I Learned Go in 3 Weeks. Yesterday, My Code Merged into k9s. From zero Go experience to a...

Kubernetes Interviews Are Broken When Trivia Matters More Than Real Skill Kubernetes interviews often fail when they test whether a candidate can recall obscure implementation details instead of showing how that person diagnoses failures, reasons through tradeoffs, and learns under pressure. Certifications can prove useful baseline knowledge, but neither a certificate nor a perfect whiteboard answer reliably proves that someone can operate a production cluster. The frustration becomes obvious when an interview demands a kernel level explanation of what happens when traffic reaches an ingress controller in a Cilium based, proxyless setup, while the actual role may involve changing a CPU request from 500m to 550m. The contrast is funny because it feels painfully familiar. Candidates prepare for architecture, networking, controllers, scheduling, and troubleshooting, then get judged on a detail they could verify in seconds during real work. That does not mean deep technical knowledge is useless. Some roles genuinely require it. The problem begins when interview difficulty becomes disconnected from job difficulty, and when memorization is treated as a shortcut for measuring engineering judgment. Why Kubernetes interview questions feel disconnected from the job The strongest complaint in the discussion was not that Kubernetes is too difficult. It was that many interview questions appear designed to establish superiority rather than measure readiness for the role. One example captured the problem perfectly: the interview asks for a detailed explanation of kernel behavior, ingress traffic, Cilium, eBPF, and proxyless networking. The work itself turns out to be a minor resource adjustment. That gap creates distrust because candidates are being filtered through a standard that the daily job may never require. A technical interview should reflect the decisions the engineer will actually make. If the job involves operating clusters, useful questions might examine how the candid

Your GitOps Hub Will Become the Bottleneck Long Before Cluster Count Tells You GitOps hub bottlenecks are usually predicted more accurately by watched object volume, reconcile queue depth, and controller memory growth than by cluster count alone. Large-scale testing described in the discussion showed Argo CD application controllers hitting out-of-memory failures around 15,000 to 20,000 cached objects per hub, while sharding and tuning delayed the limit without removing the underlying memory cost. The most important lesson was uncomfortable because it challenged the usual instinct to keep tuning the existing platform. At very large scale, architecture mattered more than configuration. Hydrated manifests helped. More replicas helped. Dynamic sharding helped. None of them changed the fact that a centralized reconciliation model still had to hold and process a huge amount of state. The testing was not presented as a universal benchmark or as proof that one tool always beats another. It was a record of one setup, built through dozens of iterations over several months, and the failures were as valuable as the successful runs. That is exactly why the results matter. They show where teams should look before the hub becomes the thing taking the fleet down. Cluster count is the wrong first metric A fleet with 1,000 tiny clusters may place less pressure on a GitOps control plane than a much smaller fleet containing thousands of applications and deeply expanded resource trees. The number of managed clusters is visible and easy to report, but it does not describe the controller’s actual workload. The more useful mental model is objects over clusters. Each application contributes desired state, live state, cached trees, reconciliation work, and queue activity. A cluster that runs a few small addons may be cheap to manage. Another cluster with many applications and large manifest sets may consume far more memory and reconciliation time. That means two fleets with the same cluster

Metabase is an open-source BI tool for building charts and dashboards over MySQL, PostgreSQL, MongoDB, Redshift, and more. This guide deploys Metabase on Kubernetes, loads the Sakila sample dataset into MySQL, builds a dashboard, and secures it behind Nginx Ingress with cert-manager TLS. Prerequisites: a Kubernetes cluster with kubectl / helm configured, a Linux workstation, a reachable MySQL server, and a domain name. Load the Sakila Sample Database Sakila models a DVD rental store — films, actors, inventory, rentals. $ sudo apt install zip -y $ wget https://downloads.mysql.com/docs/sakila-db.zip $ unzip sakila-db.zip Connect to your MySQL server (replace host/port/user): $ mysql -h <HOST_ENDPOINT> -P <DATABASE_PORT> -u <ADMIN_USER> -p mysql > CREATE DATABASE sakila ; mysql > SOURCE sakila - db / sakila - schema . sql ; mysql > SOURCE sakila - db / sakila - data . sql ; Deploy Metabase $ nano metabase.yaml apiVersion : apps/v1 kind : Deployment metadata : name : metabase spec : selector : matchLabels : app : metabase replicas : 1 template : metadata : labels : app : metabase spec : containers : - name : metabase image : metabase/metabase:latest ports : - containerPort : 3000 protocol : TCP --- apiVersion : v1 kind : Service metadata : name : metabase-svc spec : type : LoadBalancer selector : app : metabase ports : - name : http port : 8080 targetPort : 3000 Your cloud provider may need a provider-specific LoadBalancer annotation here (e.g. to set the listener protocol) — check its Kubernetes docs if the default doesn't work. $ kubectl apply -f metabase.yaml $ kubectl get deployments $ kubectl get services Wait for metabase-svc to get an EXTERNAL-IP (can take a few minutes), then visit http://<external-ip>:8080 to confirm the Metabase welcome page loads. Connect Metabase to the Database Let's get started → pick language. Enter your name, email, company, and a password. Select your use case. Database engine: MySQL . Set a display name, then host/port/database/user/pa