标签：#Kubernetes

⚡ TL;DR Four things about controller mechanics are widely half-understood: what Reconcile receives, where its work comes from, what a periodic resync is, and what a predicate turns off. I built an operator, broke it five ways, and measured each mechanism directly. The reconcile function runs in 2.71ms mean, 77/77 under 25ms , a short resync period costs zero additional API requests , and GenerationChangedPredicate cut steady-state reconciles by 48.5% without touching live repair at all. That last combination is the one that matters at scale. Repo, raw data, and harness: kirPoNik/k8s-drift-operator . 🧩 The four barriers Everyone who runs Kubernetes knows the platform repairs itself. Delete a pod, it comes back. Scale a Deployment by accident, something puts it back. Almost nobody who relies on that property can say how it works, and the gaps are specific and consequential. I keep meeting the same four: People think a controller is told what changed. It is not, and the reason it is not is the single most important design decision in Kubernetes. People think a controller polls the API server. It does not, and knowing what it does instead tells you where your API load actually comes from. People think a resync is a re-check against the cluster. It is not, which is why a short resync period is nearly free — and why the number that is expensive sits somewhere else entirely. People treat a predicate as a pure optimisation. It is a filter with a silent cost, and the cost is not the one the documentation warns you about first. So I built the smallest system that has the self-healing property, broke it on purpose ten times per failure mode, and instrumented each of those four mechanisms until I could state what it does rather than what it is said to do. What I built. One CRD called Echo , holding an image, a replica count, and a greeting. A controller keeps three child objects in sync with it — a Deployment, a Service, and a ConfigMap holding the greeting — with owner referen

I rent six dedicated servers from a company in Germany. Together they have more cores, more memory, and more SSD than most production clusters I worked on a decade ago. I run my own Kubernetes on them. Not managed. Not EKS. Not GKE. The whole stack, from the immutable OS up to the workloads. People who hear this ask me why, and the question usually arrives in one of two tones. The dangerous tone is "that's amazing, how do I do it" . The responsible tone is "why on earth would you do that to yourself" . This post is for the second group. Why the cloud is the right answer for almost everyone Let me get this out of the way honestly: by every conventional metric, I should be using the cloud. Managed Kubernetes has become genuinely good. EKS has dramatically improved over the last three years. GKE has always been better than people gave it credit for. The serverless options are mature. The serverless databases are mature. The observability is mature. The bill is predictable in the way a Tuesday is predictable. Self-hosting violates almost every assumption that makes a startup productive. Time is the most expensive resource you have. The cloud sells you abstractions that turn that time into product. Running your own substrate means the time goes into the substrate. If you are trying to ship a product to customers — go use the cloud. Stop reading this post. It will only confuse you. What the cloud does not sell you Here is what the cloud will not sell you, even if you are willing to pay extra: control over your own roadmap. The cloud's roadmap is the cloud's. They decide which APIs deprecate. They decide which regions get the new feature. They decide what your egress bill looks like. They decide whether your monitoring vendor — sitting on top of their infrastructure — is allowed to charge you eight times what it would cost you to host the same software yourself. They decide whether the small ML company hosting your fine-tuned model gets acquired by someone with very differ

If you're prepping for the Certified Kubernetes Application Developer (CKAD) exam, you've probably already found killer.sh, Killercoda, or one of the paid mock-exam platforms. I wanted something different: no account, no cloud dependency, no subscription — just a simulator that runs entirely against my own cluster, so I could rerun the same drills as many times as I wanted without worrying about usage limits. That's CKAD Dojo — free, open source, self-hosted. What it actually does 20 free mock exams, 398 questions, mapped to the official CKAD v1.35 curriculum A 120-minute countdown timer that mirrors the real exam (turns yellow at 15 min, orange at 5, red at 1) An embedded web terminal (ttyd) right next to the question panel — same gesture as the real exam UI, no window-juggling Instant, real scoring: bash functions query the actual state of your cluster against 400+ criteria, question by question. You don't have to wait until the end to know if you got it right. Runs against your own cluster — kubeadm, minikube, or kind (1.28+). Nothing leaves for the cloud. Why "dojo"? Each of the 20 practice sets is themed after a figure from Japanese mythology or the four celestial guardians (Suzaku, Byakko, Genbu, Kirin...). Resources inside each dojo follow the theme, so kubectl get pods genuinely reads like a small story instead of pod-1, pod-2, pod-3. Small detail, but it makes repeated drilling less soul-crushing. The loop Open a dojo — namespaces, workloads and Helm releases get provisioned for you. Scripts are idempotent, so you can rerun them freely. Train in the terminal — question on the left, real shell on the right, resizable divider. Arrow keys to navigate, F to flag a question, collapsible hints if you're stuck. Score whenever you want — not just at the end. Wipe and redo — read solutions.md, clean the cluster, and run the same dojo again tomorrow. The goal is reflex, not memorized answers. It's community-built 14 of the 20 dojos come from contributors — 9 as fully

Kubernetes is encouraging developers to take a closer look at KYAML, a stricter dialect of YAML designed to make Kubernetes configuration more explicit, predictable, and less prone to common YAML errors. By Craig Risi

This project started with a simple goal: run Kubernetes without keeping an EKS cluster online every day. In I Wanted Kubernetes Without an Always-On EKS Bill , I built an always-on k3s lab on my home server and proved that I could deploy, update, and roll back an application. The rollback worked, but it exposed the next problem. Kubernetes restored Version 2 while the saved YAML still declared Version 3. I corrected the file manually, but the recovery depended on repairing the running cluster and its saved instructions separately. In The Rollback Worked. My Next Deploy Could Break It Again , I designed a safer path. The automated build process would test and publish an exact image, then stop at a Git pull request. Git would record the reviewed version. Argo CD, running inside Kubernetes, would make the cluster follow that record. Now I needed to prove that the design worked outside a diagram. I followed one release from source code to running Pods. Then I tested two opposite failures: The cluster was wrong while Git was correct. Git contained a bad setting while the cluster followed it correctly. Those experiments showed both the value and the limit of GitOps. Automation can make the cluster match Git, but it cannot decide whether the human-approved version in Git is a good one. CI Built the Release but Did Not Deploy It The GitHub Actions workflow—my continuous integration, or CI, worker—ran the application tests and checked the Kubernetes package before building anything. Its job was to prove and publish a release, not to change the cluster. After validation, Buildx created a Linux AMD64 image with the full source commit baked into /version : docker buildx build \ --platform linux/amd64 \ --build-arg "APP_VERSION= $GITHUB_SHA " \ --tag " $image_name : $GITHUB_SHA " \ --provenance = mode = max \ --sbom = true \ --push \ application After publishing the image, CI read its registry digest. A digest is the image's content fingerprint: if the image changes, the digest

The official record is one sentence: an "authorized attacker," a local path traversal in Azure Kubernetes Service, 8.8 CVSS. The researchers who found the bug headlined it differently: "From AKS node root vulnerability to Microsoft Copilot hijack." Same vulnerability. The distance between those two descriptions is the story, and the aggregators missed it. Context matters here. June 2026 brought a 206-vulnerability Patch Tuesday with three disclosed zero-days, the largest on record. A "local" traversal with an EPSS of 0.00336 sinks in that noise. The two most visible public writeups are openly machine-generated, and one claims no vendor fix exists in the same entry that recommends the Microsoft update. The actual chain never got told. The chain, with the honest part labeled The flaw is CWE-22 in AKS file path handling: input is not canonicalized against a restricted base directory, so ../ sequences and absolute paths escape the intended root. The fixed line is node image build v0.20260213.5, delivered through the AKS update channel. Both facts point at Microsoft-built node-side components, not upstream Kubernetes. The load-bearing character in the CVSS vector is S:C, scope changed. The traversal is the lockpick. The scope change is the container-to-host escape. Root on a managed node hands you the kubelet's credentials, every projected service account token on the box, the runtime socket, and whatever cloud identity material the node can fetch. Worth noting: Microsoft's own title says "Remote Code Execution" while the vector says AV:L and cvefeed flatly states "Remotely Exploit: No." My read: "local" is measured from the node. An authenticated tenant running code in their own pod already holds that position. That's a normal Tuesday with a working deployment, not a high bar. Stage four, the Copilot hop, is public only as a title, and I'm flagging that instead of pretending otherwise. The shape of this attack class is standard, though. Assistants wired into control pla

Disclosure: the numbers below are from Tessera, which I work on. The reasoning applies to any proxy that sits in a session path. Every vendor page in this category says something like "scales to thousands of users". It is a useless number, because a user who is not connected costs nothing. What costs something is a session that is open right now. So here is the arithmetic instead, with the method, so you can check it against whatever you are evaluating. The unit The controller does not know how many engineers you employ and does not care how many targets are registered. It knows how many sessions are open. The planning rule that has held up for us: on a normal working day, 10–20% of a team is connected at once. A 200-person engineering organisation is 20–40 concurrent sessions, not Size for your own observed peak, but if you are estimating from scratch, start there. This matters because the difference between the two numbers is the difference between a 512 MB VM and an argument about whether you need a cluster. Memory A proxied session is mostly buffers. In our case: about 256 KB of copy buffers, plus 6 to 10 goroutines at roughly 8 KB of stack each. Call it 320 KB per session. The base Go process is about 30 MB. So 200 concurrent sessions is 200 × 320 KB ≈ 64 MB of live data, plus 30 MB base, ≈ 94 MB. Except that is not what RSS will show you, and this is the part people get wrong when they size Go services. Go does not hand memory back to the operating system promptly, and at the default GOGC=100 the collector lets the heap grow to roughly twice the live set before collecting. So resident memory settles at about double the arithmetic. Concurrent sessions vCPU Expected resident Provision up to 50 1 ~90 MB 512 MB 50–200 2 ~190 MB 1 GB 200+ 4 ~380 MB 2 GB The gap between the last two columns is headroom for spikes, not a hidden cost. A controller serving 200 sessions really does use a couple of hundred megabytes. Our Helm chart ships requests: 256Mi and limits: 1Gi ,

Disclosure: I work on Tessera, which is one of the proxy-shaped tools. Both shapes are legitimate and I try to be fair to the other one below. Most comparisons of access-control tools start with feature tables. That is the wrong end. The decision that actually determines whether a rollout finishes is the deployment shape, and there are only two. Shape one: agents and certificates You run an internal certificate authority. Hosts are configured to trust it. Users get certificates that expire in a few hours. For Kubernetes, an agent runs inside the cluster and brokers access from there. What this buys you is genuinely good. Expiry does revocation automatically, which removes the human step that fails. The credential on the user's laptop is worthless tomorrow. The model scales well because the CA does not sit in the data path — once the certificate is issued, the user talks to the target directly, so there is no proxy to size and no bandwidth to plan. What it costs is that you have to change production before you get anything. sshd_config gets rewritten across the estate to add TrustedUserCAKeys . An agent gets deployed into every cluster. In some setups the tool's binary is copied onto hosts. None of that is technically hard. It is organisationally hard. You need a change window, sign-off from whoever owns those hosts, and a rollback plan — for a project whose entire benefit is "nothing bad will happen later". That conversation is where access-control rollouts stall, and it stalls most reliably in exactly the organisations that need the tool most: the ones where nobody is quite sure who owns which box. The other cost is that the CA private key becomes the most sensitive object your company owns, and now you operate a CA. Shape two: a proxy The credential stays on a controller. The user authenticates to the controller. The controller opens its own connection to the target, authenticates with the real credential, and relays. The target sees a normal connection from a nor

In eleven days, on 11 September 2026, the reporting obligations of the EU Cyber Resilience Act start applying to anyone who puts a product with digital elements on the European market. Not the full regulation. Just the part where, if you find out an actively exploited vulnerability is in your product, you have 24 hours to tell ENISA about it. I have watched a lot of engineering teams meet this class of deadline for the first time. It usually goes the same way. Somebody in sales gets a security questionnaire. Somebody in engineering gets forwarded the questionnaire. Three weeks later there is a shared folder called evidence-final-v3 with 200 screenshots in it, and nobody can tell you which screenshot answers which question. I have spent the last several months building a tool whose entire job is that folder, so I read the instruments properly. This is what I found out. It is written for engineers, not for a compliance team, and I try to be specific about what the text says rather than what a vendor blog says it says. Where SOC 2 came from, and why that still shapes it SOC 2 exists because of a misuse. In 1992 the AICPA published SAS 70, an auditing standard for service organisations. Its purpose was narrow: if you outsourced your payroll, your auditor needed some assurance that your payroll provider's internal controls did not corrupt your financial statements. It was an accounting instrument, for accountants, about financial reporting. Then the industry outsourced everything else. By the mid-2000s companies were sending their customer data to service providers, and they wanted assurance about that , not about financial reporting. There was nothing designed for it, so they asked for the thing that existed. Vendors started waving SAS 70 reports around as proof they were secure. They were not proof of that. SAS 70 had no defined control set at all: the service organisation wrote its own control objectives, and the auditor tested against whatever had been written. Two S

Hey all 👋 Last episode a power cut exposed an uncomfortable fact: my cluster's entire memory lived in one SQLite-flavored database, on one laptop, bound to one Wi-Fi address, guarded by one aging battery. Four single points of failure in a trench coat. The fix isn't making that database unkillable. The fix is making it unimportant . If every manifest lives in git and something reconciles the cluster against git continuously, then "the datastore died" stops being a tragedy and becomes a reboot with extra steps. So: Flux . Here's the setup, and the four ways I face-planted installing it. 🗂️ The Shape of the Repo clusters/homelab/ flux-system/ <- Flux writes this at bootstrap; hands off infrastructure.yaml <- points at infrastructure/ apps.yaml <- points at apps/ infrastructure/ controllers/ <- Longhorn + Envoy Gateway HelmReleases configs/ <- GatewayClass, Gateway, StorageClasses apps/homelab/ airflow/ <- the actual point of all this Three Flux Kustomizations, chained: infra-controllers → infra-configs → apps , via dependsOn . That chaining is not decoration. My GatewayClass can't exist until Envoy Gateway's CRDs exist, and the CRDs arrive with the controller's Helm chart. Without dependsOn , Flux sprints ahead, tries to create a GatewayClass into a cluster that's never heard of GatewayClasses, and fails with the enthusiasm of a golden retriever running into a glass door. dependsOn plus wait: true turns that into: install controllers, wait until healthy , then configs, then apps. Boring. Sequential. Correct. The three great virtues. 🔑 Sidequest 1: The Token Bureaucracy flux bootstrap github needs a GitHub token, and the docs-diving summary is: Classic PAT: repo scope. Needed if Flux should create the repo. Fine-grained PAT (pre-created repo): Contents read/write, Metadata read, and — the one everyone misses — Administration read/write , because Flux installs an SSH deploy key on the repo, and deploy keys are an admin operation. Here's the nice part: the deploy key is

After development comes deployment - whether on-premise or on a cloud based environment. And then we face a simple question: how much resource should I assign to this system? What is the ideal numbers? If we get this wrong, we often need to go back time and again to fine tune - either to ensure our application is capable of handling the targeted load, or to avoid paying for resources we are not using. This article explains the approach step by step. So that we spend just enough time upfront to avoid spending exponentially more time and money at later stages. Who Is This For? This article is written primarily for developers. But if you are a manager or a CTO, there are sections written specifically for you. Feel free to jump straight there. 👉 If you are a Manager or Project Manager 👉 If you are a CTO or Architect For everyone else - the full article is worth reading top to bottom at least once. But if you are revisiting a specific topic, jump to whatever is relevant. Table of Contents Local is the Starting Point When Should You Start Thinking About Right Sizing? How Long Will This Actually Take? Start With What You Have - Your Local Setup Setting Up Your Load Generation - The Hammer The Cost of Testing - This Is Not Free Sizing Your Pod More Resources Per Pod or More Pods? Scaling - Easy to Set Up, Hard to Get Right Periodic Right-Sizing - You Are Not Done Yet Local is the Starting Point Local system is always where we start. To try things out, to check if things work. But 99% of what we test locally is the sunny day scenario. Does the MVP work? Does the happy path hold? Even if you're diligent enough to test negative scenarios, you're almost certainly not testing production-level load on your laptop. Which means you have no idea what resources your app actually needs when it matters . This is where the problem starts. On local, we routinely kill the heavy IDE, close browser tabs, shut down background processes -without ever stopping to ask: how much memory and CPU d

This article is Part of a series on running vLLM on AKS and walks through creating an AKS cluster with a GPU node pool, deploying vLLM onto it, and wiring up Prometheus and Grafana for visibility. Companion pieces: Choosing the right GPU | Why your autoscaler flaps | Source Setup Summary Cloud: Azure GPU node: Standard_NV36ads_A10_v5 (1× A10, 24 GB) Image / model: vllm/vllm-openai:latest serving Qwen/Qwen2.5-7B-Instruct-AWQ Observability: kube-prometheus-stack (Prometheus + Grafana), KEDA, NVIDIA DCGM exporter All commands below are bash. The steps are ordered and each one depends on the previous. Dependency chain The build order follows one chain: model → VRAM requirement → GPU SKU → region availability → quota. Step 0 — Prerequisites (one-time, survives resource group deletion) GPU quota. Request through Portal → Quotas → Compute → This article: Requested Standard NVADSA10v5 Family vCPUs = 108 in westus (108 = 3 nodes × 36 vCPUs, matching the autoscaler's max-count 3 set in step 3). Quota is granted per-subscription and survives resource group deletion, so this step happens once, not on every rebuild. A quota is Azure's per-subscription limit on how much of a resource (here, GPU vCPUs in a specific VM family) you're allowed to provision at once. New subscriptions start at 0 for GPU families since it's expensive and can be abused. You need it because without an approval, az aks nodepool add for a GPU will fail outright. The request goes through manual Azure approval, so it has to happen before you plan to build. * Prerequisites * Existing Azure Subscription: Local tooling: Helm 3+ kubectl Bash Bash Variables to set for use through the setup RG = <resource-group-name> CLUSTER = <cluster-name> LOCATION = <preferred-location> Step 1 — Create Resource group az group create -n $RG -l $LOCATION Step 2 — Create AKS cluster, on a CPU system pool az aks create -g $RG -n $CLUSTER \ --node-count 1 --node-vm-size Standard_D2s_v5 \ --generate-ssh-keys The GPU does not go on thi

I'm fairly new to SRE/DevOps, and one of the topics I recently spent time studying properly was EKS cluster upgrades . My first instinct, like most people starting out, was: "it's just a version bump, click upgrade in the console, done." That's basically what most beginner blog posts say too. But the more I read and the more I dug into real-world postmortems and discussions, the more I realized — the actual Kubernetes control plane upgrade is the easy part. Almost everything that can go wrong seems to happen around it, not because of it. Sharing what I learned here, mainly for my own notes, but hoping it's useful for anyone else early in their journey too. Learning #1: There's No "Undo" Button This was the first thing that surprised me. I assumed upgrades work like most software — if something breaks, you roll back. But with EKS, you cannot downgrade the control plane version once you upgrade it. So the plan can't be "upgrade, and if it breaks, revert." It has to be "test enough beforehand that breaking isn't really an option," and if something does go wrong, the fix is always moving forward, not backward. That single fact changes how you're supposed to approach the whole thing — testing has to happen before the button is clicked, not after. Learning #2: APIs Get Deprecated, and It's Usually Not Your Own Code That Breaks Kubernetes removes old API versions on a schedule. I already knew this conceptually, but what I didn't realize is that the risk usually isn't your own YAML files — it's the Helm charts and third-party tools you installed a while back and forgot about , which might still be using an older API version internally. There are tools built exactly for catching this before it becomes a problem: pluto detect-helm -owide pluto detect-files -d ./manifests kubent (kube-no-trouble) does something similar. I hadn't heard of either tool before researching this, and it made me realize how much of "being good at Kubernetes" is really just knowing which small tools e

Microsoft is placing greater emphasis on controlling disruption in Azure Kubernetes Service (AKS) Node Auto-Provisioning (NAP), publishing new guidance to help platform teams balance the efficiency benefits of automated node consolidation with application availability. By Craig Risi

Scalable Guardrail Service ASP.NET Core Kubernetes: Architecture, Code, and Ops Quick Answer Scalable Guardrail Service ASP.NET Core Kubernetes: A dedicated ASP.NET Core guardrail microservice on Kubernetes validates LLM requests, enables instant policy updates via Redis, and scales with custom HPA for high‑throughput. Scalable Guardrail Service ASP.NET Core Kubernetes: Why a Dedicated Guardrail Microservice Matters When you expose an LLM‑powered API to the world, every request is a potential compliance risk. A single malformed prompt can surface PII, trigger a policy violation, or even cause a brand‑damaging output. In my experience, the first version of such a system is a set of ad‑hoc filters sprinkled across controllers. Under load, those filters become latency bottlenecks, policy updates race, and audit trails vanish. The root cause is a missing architectural layer that treats guardrails as a first‑class microservice that can scale horizontally, be updated live, and be observed independently. Guardrail Layer Requirements We need a guardrail layer that: Validates every request before it hits the LLM engine. Can be updated without redeploying the entire API surface. Provides per‑tenant isolation and versioning. Logs every decision for compliance and red‑team analysis. Runs at the same scale as the LLM inference service. When This Fails in Production Policy updates are applied via a shared ConfigMap and the pods do not reload, so new rules are never enforced. The guardrail service is single‑instance; a spike in requests triggers a queue that exceeds the LLM engine’s rate limit, causing a cascading failure. Audit logs are written to local disk; a pod crash loses events. Latency spikes because each request performs a synchronous Redis lookup for every policy. Common Mistakes Engineers Make Embedding guardrail logic inside the API controller rather than a dedicated middleware. Using in‑memory policy caches without a TTL, leading to stale rules. Ignoring the fact that

Upgrading Kubernetes clusters across a large enterprise fleet is often a balancing act between staying current with security patches and avoiding outages. By default, Google Kubernetes Engine (GKE) rolls out automatic upgrades progressively according to Google Cloud regional timelines. While regional rollout works well for standalone clusters, it does not understand your organization's business topology. If you run staging clusters in us-central1 and critical production clusters in us-east1 , a standard regional rollout could upgrade your production environment before your pre-production validation completes. The General Availability (GA) release of GKE rollout sequencing with custom stages solves this challenge. It provides platform teams with declarative control to sequence cluster upgrades across fleets, environments, and even distinct Google Cloud organizations according to business criticality rather than cloud geography. How rollout sequencing works Rollout sequencing builds on GKE fleet management. Fleets serve as logical boundaries for environments such as development, staging, and production. With rollout sequencing, you define an ordered pipeline of upgrade stages managed by a central resource called RolloutSequence . When GKE publishes a new automatic upgrade target for a release channel, or when you explicitly trigger a target version, the system creates a Rollout object. This rollout progresses through your defined stages sequentially: Control plane upgrades start in the first stage. Once all control planes in that stage reach the target version, a stage soak timer begins. Node upgrades run in parallel with control plane upgrades, respecting node pool upgrade strategies such as surge or blue-green. When both control planes and nodes complete their upgrade and satisfy the configured soak duration, the rollout advances to the next stage in the sequence. If an individual stage contains clusters that take longer than 30 days to finish upgrading—due to restr

Short answer: for a Node.js app in Docker or Kubernetes, give startup, readiness, and liveness probes separate meanings, keep routine health traffic out of application logging, and measure state transitions instead of counting every successful check. For a property-management API rolling out a new pricing rule, this preserves useful metrics: whether an instance can calculate rent correctly and accept traffic, without turning each kubelet poll into noise. Which health signal should control each container decision? Start with the decision, not the endpoint name. Signal Question it answers Include Exclude Action Startup Has initialization completed? Configuration parsing, pricing-rule compilation, required local warm-up Long-term dependency health Allow the process more time before other probes apply Readiness Can this instance safely receive a new pricing request now? Ability to serve the active rule version and any required dependency state Optional analytics and background exports Remove the pod from Service endpoints Liveness Is the process stuck beyond local recovery? Event-loop progress or another narrow process invariant Database, cache, and third-party availability Restart the container This split is the main noise filter. A downstream dependency becoming unavailable can make a pod unready, but restarting the same healthy process usually doesn't repair that dependency. If the dependency is placed in liveness anyway, every pod can restart together. The health response has then amplified one problem into two: lost capacity plus a restart storm. The pricing rollout makes readiness more demanding than “the port is open.” Imagine rule version rent-2026-08 is enabled for one building cohort. A newly started instance has loaded configuration but hasn't compiled that version yet. It is alive. It isn't ready. Its startup check should hold back liveness and readiness until initialization finishes; afterward, readiness should stay false until the active rule can be evalua

By combining ProgressivePerGroup with Placement decision groups, you can roll out add-on configuration changes in the order dev → stg → prod. In this article, I use cluster-proxy as an example to explain the required configuration and how the rollout actually works. Overview flowchart TB Upgrade["helm upgrade<br/>change tag to vX.Y.Z"] subgraph Hub["Hub cluster"] direction TB Manager["cluster-proxy-addon-manager<br/>update Deployment"] Config["ManagedProxyConfiguration<br/>update spec"] ProxyServer["proxy-server<br/>update Deployment"] Hash["proxyAgent config<br/>update spec hash"] Rollout["OCM add-on manager<br/>ProgressivePerGroup"] Groups["progress through decision groups<br/>dev → stg → prod<br/>success + minSuccessTime before next group"] AddOn["ManagedClusterAddOn in current group<br/>Configured=True"] Render["cluster-proxy manager<br/>render agent chart"] Work["update ManifestWork"] end subgraph Spoke["spoke clusters in the current group"] direction TB WorkAgent["work-agent"] ProxyAgent["proxy-agent<br/>update Deployment"] end Upgrade -->|Helm updates directly| Manager Upgrade -->|Helm updates directly| Config Config -->|proxyServer.image<br/>not part of rollout| ProxyServer Config -->|proxyAgent.image<br/>part of rollout| Hash Hash --> Rollout Rollout --> Groups Groups -->|current group only| AddOn AddOn --> Render Render --> Work Work --> WorkAgent WorkAgent --> ProxyAgent WorkAgent -.->|Applied / Available| Work Work -.->|hash matches + Ready| Rollout classDef immediate fill:#fff3cd,stroke:#a66b00,color:#332200; classDef staged fill:#e8f3ff,stroke:#2563a6,color:#102a43; classDef spoke fill:#eaf7ed,stroke:#2f855a,color:#173d2a; class Manager,Config,ProxyServer immediate; class Hash,Rollout,Groups,AddOn,Render,Work staged; class WorkAgent,ProxyAgent spoke; Yellow indicates updates that happen immediately on the Hub. Blue indicates updates controlled by ProgressivePerGroup , and green indicates processing on the spoke clusters. Dashed lines represent status r

We have a web server container running the nginx image. The access and error logs generated by the web server are not critical enough to be placed on a persistent volume. However, Nautilus developers need access to the last 24 hours of logs so that they can trace issues and bugs. Therefore, we need to ship the access and error logs for the web server to a log-aggregation service. Following the separation of concerns principle, we implement the Sidecar pattern by deploying a second container that ships the error and access logs from nginx. Nginx does one thing, and it does it well - serving web pages. The second container also specializes in its task - shipping logs. Since containers are running on the same Pod, we can use a shared emptyDir volume to read and write logs. Create a pod named webserver . Create an emptyDir volume named shared-logs . Create a regular container in the webserver pod from the nginx:latest image named nginx-container , and an init container from the ubuntu:latest image named sidecar-container . Add the following command to the sidecar-container "sh","-c","while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log; sleep 30; done" Mount the shared-logs volume in both containers at /var/log/nginx . Ensure all containers are in a running state. What is a Sidecar Container? Think of a sidecar like a motorcycle sidecar – it's attached to the main vehicle and extends its capabilities without changing the main vehicle itself. ┌─────────────────────────────────────────────────────────────────────────────┐ │ The Sidecar Analogy │ │ │ │ ┌────────────────────────────────────────────────────────────────────────┐ │ │ │ Motorcycle: The Main Vehicle │ │ │ │ - Does its primary job (serving web pages) │ │ │ │ - Doesn't worry about extra tasks │ │ │ └────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────────────────┐ │ │ │ Sidecar: Adds Extra Functional

Control Plane (Master) & Worker Nodes Control Plane components: API Server Scheduler Control Manager etcd Worker Node components: Container Runtime Kubelet Kube-proxy Node Processes Each node has multiple Pods on it. 3 processes must be installed on every node — used to schedule and manage those Pods. Nodes are cluster services that actually do the work. Container Runtime Examples: Docker, containerd, CRI-O. containerd is used in worker nodes — it's lightweight in nature. This should be installed on every node because application Pods need to run containers inside the node. Kubelet The process which schedules the Pods and containers underneath is Kubelet. Kubelet interacts with both the container and the node. Kubelet starts the Pod with the container inside. Communication between two nodes is because of Services. Creation of Pod: Kubelet insures the Pod is always running — if not, it will inform etcd. Kube-proxy Kube-proxy forwards the request from Pod to Service. Makes use of the communication, with load balancing. Provides networking (container ID, IP address). Load balancing — basically using IP tables. It makes sure to send the request to the same machine instead of sending it to others (from same node communications). So, how do you interact with this cluster? Schedule the Pod Monitor Re-schedule/restart the Pod Join a new node Managing processes are done by master nodes (the control plane). API Server When you, as a user, want to deploy a new application in a Kubernetes cluster, you interact with the API server using some client — could be UI or CLI. It's a cluster gateway — it gets the initial request of any update into the cluster, even the queries from the cluster. It also acts as gatekeeper for authentication. It means when you want to schedule new Pods, deploy new applications, create new services, or any other components — you have to talk to it first. Flow: Some request → API server → Validates request → Other processes → Pods Only one entry point to t