标签：#DevOps

Google Vertex AI is Google Cloud's managed ML platform with experiment tracking, training jobs, pipelines, a model registry, and endpoints, but it locks you into GCP-specific APIs and per-use billing. ClearML is an open-source MLOps platform that covers the same ground on any Docker host or Kubernetes cluster, capturing training metrics automatically with no code changes and keeping every byte of data on infrastructure you control. This guide deploys ClearML Server with Docker Compose and Traefik, registers an agent, runs an experiment, builds a pipeline, runs a hyperparameter sweep, and deploys a Triton-served model. By the end, you'll have a self-hosted Vertex AI replacement covering the full ML lifecycle. Vertex AI → ClearML Mapping GCP Vertex AI ClearML Equivalent Purpose Vertex AI Workbench ClearML Web UI Browser-based monitoring and configuration Vertex AI Experiments ClearML Experiment Manager Automatic hyperparameter/metric/artifact tracking Vertex AI Training Job ClearML Agent + Tasks Any machine becomes a remote worker via queues Vertex AI Pipelines ClearML Pipelines Python-native DAGs, no separate compile step Vertex AI Model Registry ClearML Model Repository Versioned models with full lineage Vertex AI Endpoints ClearML Serving (Triton) Self-hosted inference with canary/A-B support Cloud Monitoring/Logging ClearML Scalars/Plots Built-in metrics and hardware dashboards Prerequisite: Ubuntu host with Docker + Compose, DNS A records for app.clearml.example.com , api.clearml.example.com , files.clearml.example.com . NVIDIA Container Toolkit if you'll run GPU agents. Deploy the ClearML Server 1. Raise Elasticsearch's virtual memory limit and restart Docker: $ echo "vm.max_map_count=524288" | sudo tee /etc/sysctl.d/99-clearml.conf $ sudo sysctl --system $ sudo systemctl restart docker 2. Create persistent data directories with the correct ownership: $ sudo mkdir -p /opt/clearml/ { data/elastic_7,data/mongo_4/db,data/mongo_4/configdb,data/redis,data/fileserver,

Matomo is an open-source web analytics platform that keeps full ownership of visitor data on infrastructure you control, a privacy-first alternative to Google Analytics. This guide deploys Matomo using Docker Compose with a MariaDB backend, Nginx as the entry point, and Certbot issuing the TLS certificate before the stack goes live. By the end, you'll have Matomo tracking a website with a signed HTTPS certificate at your domain. Prepare Docker $ sudo usermod -aG docker $USER $ newgrp docker Set Up the Project 1. Create the project directory: $ mkdir matomo && cd matomo 2. Open the firewall: $ sudo ufw allow 80/tcp $ sudo ufw allow 443/tcp 3. Create the environment file: $ nano .env MYSQL_ROOT_PASSWORD = your_strong_mysql_root_password MYSQL_PASSWORD = your_strong_mysql_matomo_password Use passwords of at least 16 characters. 4. Create the Nginx config: $ mkdir nginx $ nano nginx/matomo.conf server { listen 80 ; server_name matomo.example.com ; location /.well-known/acme-challenge/ { root /var/www/certbot ; } location / { return 301 https:// $host$request_uri ; } } server { listen 443 ssl http2 ; server_name matomo.example.com ; ssl_certificate /etc/letsencrypt/live/matomo.example.com/fullchain.pem ; ssl_certificate_key /etc/letsencrypt/live/matomo.example.com/privkey.pem ; location / { proxy_pass http://app:80 ; proxy_set_header Host $host ; proxy_set_header X-Real-IP $remote_addr ; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for ; proxy_set_header X-Forwarded-Proto $scheme ; } } Replace every matomo.example.com occurrence with your domain — it appears in both server blocks and the certificate paths. Deploy with Docker Compose $ nano docker-compose.yaml services : db : image : mariadb:11.4 command : --max-allowed-packet=64MB restart : always volumes : - matomo-db-data:/var/lib/mysql environment : - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD} - MYSQL_DATABASE=matomo - MYSQL_USER=matomo - MYSQL_PASSWORD=${MYSQL_PASSWORD} networks : - matomo-net app : image

Plausible Analytics is an open-source, self-hosted web analytics tool that tracks traffic without cookies or personal data collection, a privacy-first alternative to Google Analytics. This guide deploys Plausible Community Edition using Docker Compose, fronts it with Nginx, and secures it with a Let's Encrypt certificate. By the end, you'll have Plausible tracking a website's traffic securely at your domain. Configure the Environment 1. Clone the Community Edition repo: $ mkdir ~/plausible $ cd ~/plausible $ git clone https://github.com/plausible/community-edition.git $ cd community-edition 2. Generate a secret key: $ openssl rand -base64 64 | tr -d '\n' 3. Create the environment file: $ nano .env ADMIN_USER_EMAIL = admin@example.com ADMIN_USER_NAME = admin ADMIN_USER_PWD = ADMIN_PASSWORD BASE_URL = https://plausible.example.com SECRET_KEY_BASE = YOUR_SECRET_KEY_BASE DATABASE_URL = postgres://postgres:postgres@plausible_db:5432/plausible_db CLICKHOUSE_DATABASE_URL = http://plausible_events_db:8123/plausible_events_db Fill in your email, a strong admin password, your domain, and the secret key generated above. 4. Expose the app port via a Compose override (auto-merged with compose.yml ): $ nano compose.override.yaml services : plausible : ports : - 127.0.0.1:8000:8000 Deploy with Docker Compose $ docker compose up -d $ docker compose ps Confirm the app, Postgres, and ClickHouse (events DB) containers are all running. Front with Nginx and Let's Encrypt 1. Install Nginx: $ sudo apt update $ sudo apt install nginx -y 2. Create the virtual host: $ sudo nano /etc/nginx/sites-available/plausible.conf server { listen 80 ; listen [::]:80 ; server_name plausible.example.com ; access_log /var/log/nginx/plausible.access.log ; error_log /var/log/nginx/plausible.error.log ; location / { proxy_pass http://127.0.0.1:8000 ; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for ; proxy_http_version 1.1 ; proxy_set_header Upgrade $http_upgrade ; proxy_set_header Connection "Upgr

Part 1: Solving GOP Structure and Compatibility Issues Operating a digital television network isn't just about keeping channels on air—it's about maintaining quality that viewers expect and troubleshooting issues before they escalate. When something goes wrong in live broadcasting, every second counts. But how do you quickly pinpoint whether the problem lies in encoder settings, transport stream structure, or temporal metadata? This is where specialized stream analysis tools become essential. In this series of articles, we'll walk through real-world scenarios that broadcast engineers face daily and show practical approaches to diagnosing and resolving them. When File Analysis Becomes Critical While live monitoring catches issues as they happen, file-based analysis is your diagnostic microscope. Here's the typical workflow: something breaks in production, engineers capture a few minutes of the problematic stream, and now they need to understand exactly what went wrong. File analyzers serve three primary purposes: Troubleshooting: Identifying the root cause of broadcast issues Encoder optimization: Fine-tuning compression settings Quality control: Validating compliance with standards and specifications Let's explore how this works in practice with actual tools and techniques. The GOP Structure Problem Here's a scenario every broadcast engineer has encountered: legacy set-top boxes or older TV models suddenly can't play your stream. The audio works, video starts and stops, or you see freezing. The culprit? Often, it's the GOP (Group of Pictures) structure. H.264 has been around since 2003—over 20 years. Almost everything supports it, yet you'll still find legacy equipment that struggles with certain configurations. Specifically, the number of B-frames can make or break compatibility. Why B-frames matter: They enable lower bitrates while maintaining quality by increasing encoding complexity through bidirectional prediction. But this comes at a cost—a more complex refere

Modern software development thrives on rapid iteration. Organizations deploy new features, bug fixes, and infrastructure updates multiple times each day to remain competitive and respond quickly to customer needs. Continuous Integration and Continuous Delivery (CI/CD) have transformed software delivery by automating repetitive tasks and accelerating release cycles. However, speed without security creates significant risk. A fast deployment pipeline that introduces vulnerable code into production can expose organizations to data breaches, service disruptions, and compliance violations. Conversely, excessive manual security reviews can slow innovation and delay valuable releases. The solution lies in integrating security directly into the CI/CD pipeline rather than treating it as a separate checkpoint. This philosophy, commonly known as DevSecOps, enables organizations to deliver software rapidly while maintaining a strong security posture. Understanding CI/CD Pipelines What Is Continuous Integration? Continuous Integration (CI) is the practice of frequently merging code changes into a shared repository. Every commit automatically triggers builds and tests, allowing development teams to identify integration issues early instead of waiting until the end of a project. Frequent integration encourages collaboration, reduces merge conflicts, and improves overall software quality. What Is Continuous Delivery? Continuous Delivery extends Continuous Integration by ensuring that validated code is always in a deployable state. Automated testing, packaging, and release preparation make it possible to deploy new versions with minimal manual effort whenever the business is ready. What Is Continuous Deployment? Continuous Deployment goes one step further by automatically releasing approved changes to production once they pass all quality and security checks. This approach significantly shortens release cycles while requiring a high level of confidence in pipeline automation. Benefi

How to Monitor Website Changes Automatically I run a few websites and need to know immediately when something breaks. A CSS regression, a broken layout, a missing section. Manual checking doesn't scale, and text-based monitoring misses visual issues. The {{screenshot-diff}} on Apify takes two screenshots and produces a pixel-level comparison with an overlay showing exactly what changed. How It Works Take a baseline screenshot of the correct state. Then take a current screenshot of the live page. The actor compares pixel by pixel and returns a diff image with changed pixels highlighted, plus a percentage telling you how much changed. import requests , time API_TOKEN = " YOUR_APIFY_TOKEN " def capture_screenshot ( url ): resp = requests . post ( " https://api.apify.com/v2/acts/weeknds~website-screenshot-api/runs " , headers = { " Authorization " : f " Bearer { API_TOKEN } " }, json = { " url " : url , " fullPage " : True } ) run_id = resp . json ()[ " data " ][ " id " ] time . sleep ( 15 ) items = requests . get ( f " https://api.apify.com/v2/acts/weeknds~website-screenshot-api/runs/ { run_id } /dataset/items " , headers = { " Authorization " : f " Bearer { API_TOKEN } " } ). json () return items [ 0 ][ " screenshotUrl " ] def compare_screenshots ( baseline_url , current_url ): resp = requests . post ( " https://api.apify.com/v2/acts/weeknds~screenshot-comparison-tool/runs " , headers = { " Authorization " : f " Bearer { API_TOKEN } " }, json = { " baselineImageUrl " : baseline_url , " currentImageUrl " : current_url , " threshold " : 0.01 } ) run_id = resp . json ()[ " data " ][ " id " ] time . sleep ( 10 ) items = requests . get ( f " https://api.apify.com/v2/acts/weeknds~screenshot-comparison-tool/runs/ { run_id } /dataset/items " , headers = { " Authorization " : f " Bearer { API_TOKEN } " } ). json () return items [ 0 ] baseline = capture_screenshot ( " https://mysite.com " ) current = capture_screenshot ( " https://mysite.com " ) result = compare_screenshots ( b

Amazon Web Services (AWS) has announced a major expansion of its AWS DevOps Agent, introducing new release management capabilities designed to assess code changes and autonomously test software before it reaches production. By Craig Risi

We build an internal helpdesk, and I want to talk through a problem we only partly solved — because I suspect a lot of you have hit it too, and I'd genuinely like to hear how you handled it. The most requested thing from our users was never "better ticket forms." It was "please make the duplicates stop." Here's the shape of it. A deploy goes slightly wrong at a 40-person company. Within ten minutes you have: a handful of chat messages : "login is broken", "can't get into dashboard???", "deploy looks weird" several error-tracker events (whatever you run — Sentry, Rollbar, an APM): TokenExpiredError ×2, a 401 spike on /api/auth , a 5xx spike on auth-svc a couple of emails to IT : "access token expired", "need login reset" Nine items across three channels. One root cause: token rotation broke in that deploy. Whoever's on rotation spends the morning proving that, instead of fixing anything. We wanted to automate the recognition step — "these are the same thing" — not the fixing step. This is the honest version: what we tried, the small thing we actually shipped, and the parts we haven't cracked. If you've built something similar, I'd love to be told what we got wrong. Attempt 1: rules and keywords (broke immediately) The obvious first cut: normalize ticket text, match on keywords and categories, merge on high overlap. It fails on the example above, and it fails structurally: "login is broken" and TokenExpiredError share zero tokens. The human on rotation isn't string-matching — they know a deploy just happened, they know what auth-svc does, they've seen this failure shape before. Rules encode none of that. Rule systems also rot. Every incident teaches you a new synonym for "it's down," and six months in you own a regex museum nobody wants to touch. Maybe you've kept one of these healthy long-term — if so I'd honestly like to know how. Attempt 2: embed everything, cluster by similarity (the one we didn't ship) The tempting next move: embed ticket text, cluster on cosine

SaaS software vendor HubSpot has described how its semantic search platform grew from a proof of concept into an internal service that now manages more than 20 billion vectors across 38-plus teams. The company says the system now supports agents, RAG, and contact deduplication, and that the increase in agent usage has made retrieval quality and latency more important than before. By Matt Saunders

Quick Note In my previous article, I mentioned that Linux Troubleshooting Flow for Beginners would be the final post in this series. While preparing it, I realized there were a few practical Linux skills every beginner should learn first. These topics will make the troubleshooting guide much easier to understand and follow. Before we wrap up the series, we'll cover: Package Management Finding Files & Text Viewing Files Efficiently File Compression Then we'll bring everything together in the final Linux Troubleshooting Flow for Beginners. Introduction Installing software on Linux is very different from Windows. On Windows, you usually download an .exe installer. On Linux, software is typically installed and managed using package managers . This is one of the most practical skills every Linux beginner should learn early. What is a Package? A package is a ready-to-install bundle that contains: The main program Required libraries Configuration files Documentation Examples: nginx , git , docker , curl , vim Think of a package as a ready-to-install software box. What is a Package Manager? A package manager is a tool that installs, updates, removes, and manages software packages. Instead of downloading software manually, you simply run a command. Example: sudo apt install git The package manager automatically: Downloads packages from trusted repositories Install required dependencies automatically Upgrade installed software Removes them cleanly Instead of manual downloading, you just run one command. Why Use a Package Manager? Without package managers, you would have to: Search for software manually Download files from websites Install dependencies yourself Update each application separately Package managers automate all of this. What is a Repository? Package managers download software from repositories. A repository is a trusted online collection of software packages maintained by your Linux distribution. Instead of downloading software from random websites, Linux install

Test Isolation: A Lesson I Learned While Migrating Playwright Tests During my software engineering internship, I helped optimize our CI pipeline by identifying which E2E tests could safely run in parallel. That work quickly taught me that the biggest obstacle wasn't Playwright or Python, it was test isolation. This article is about that lesson. What is test isolation? A simple rule I now use is this: if a test can't run by itself with the same outcome, it probably isn't truly isolated. A well-isolated test should produce the same result whether it: runs by itself runs first or last runs after another test runs in parallel with hundreds of other tests To understand test isolation, it also helps to understand what state means. State isn't limited to database rows. During the migration, I found tests interacting with many different kinds of state. database records global configuration filesystem resources application caches If any of these are shared between tests, they become potential sources of hidden dependencies. How tests lose isolation As I started reading the existing test suite, I noticed a recurring pattern. Many tests assumed something about the environment instead of creating it themselves. Some expected specific data to already exist. Others modified global settings without restoring them afterward. Some searched for rows based on their position in a table instead of using a stable identifier like a name or ID. None of these looked particularly problematic when reading a single test. The problems only appeared once the entire suite started running together. One test would leave behind data another test didn't expect. A shared configuration would silently affect unrelated tests. A UI assertion would suddenly fail because another test inserted an extra row into the same table. Individually, the tests appeared independent. Together, they formed hidden dependencies. Not all shared state is equally difficult to isolate One realization that helped me reason abou

Multi stage builds sao uma das melhores features do Docker para manter imagens pequenas e organizadas. Vou mostrar como aplicar isso em um projeto Python real. Crie um arquivo app.py simples: # app.py def main(): print("Hello from a multi stage build") if __name__ == "__main__": main() Agora crie o Dockerfile sem multi stage: FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD ["python", "app.py"] Essa imagem inclui o pip, o cache do pip e ferramentas de build que nao precisamos em producao. O resultado e uma imagem maior que o necessario. Com multi stage builds separamos o ambiente de build do ambiente final. Veja o mesmo Dockerfile com dois stages: FROM python:3.12-slim AS builder WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt FROM python:3.12-slim WORKDIR /app COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages COPY . . CMD ["python", "app.py"] O primeiro stage instala as dependencias. O segundo stage copia so o que importa. O resultado e uma imagem final muito menor. Para construir e ver o tamanho: docker build -t minha-app . docker images | grep minha-app Para linguagens compiladas como Go o ganho e ainda maior. Veja um exemplo com uma aplicacao Go: FROM golang:1.23 AS builder WORKDIR /app COPY go.mod go.sum ./ RUN go mod download COPY . . RUN CGO_ENABLED=0 GOOS=linux go build -o /app/server FROM scratch COPY --from=builder /app/server /server CMD ["/server"] A imagem final comeca do zero (scratch). Nao tem shell, sistema operacional, nem ferramentas de build. So o binario compilado. Uma dica pratica: sempre nomeie seus stages com AS para facilitar a leitura. Use nomes como builder, test, ou dev. Isso ajuda a saber o que cada stage faz sem precisar contar linhas. That's all for now. Thanks for reading!

A while back I wrote about 64% of our merged PRs being written by AI . A few people (reasonably) asked: "nice story, but can I verify any of that?" So we put it on a public, auto-updating dashboard: 👉 www.codens.ai/stats/en It shows, for our GitHub organization: What % of merged PRs are authored by AI agents (currently 65%) Median time from PR-open to merge (2 minutes) Who merged what — task-execution agents vs maintenance bots vs auto-fix vs humans Weekly AI-merge counts and a per-repository breakdown Every number is tallied straight from the GitHub API by a small collector script, and the page regenerates itself weekly. We can't inflate it — it's measured, and the down weeks show up too (that's kind of the point). Why bother making it public Two reasons. 1. "Trust me bro" doesn't scale. When you claim most of your code is AI-written, the only honest move is to expose the raw counts so anyone can sanity-check them. The dashboard is that receipt. 2. It's a live dogfooding test. The whole thing — the PRD, the implementation, the review, the auto-fix on production errors — runs on Codens , our own AI dev-automation suite. If our own numbers ever tanked, the dashboard would be the first place it'd show. Public accountability is a good forcing function. Funny footnote: the dashboard itself was code-reviewed by our own AI reviewer before it shipped, and it caught two real bugs — an OG-image percentage that had drifted out of sync with the live number, and a broken error-fallback that would have blanked the page on malformed data. The tool built to sell "AI reviews your PRs" reviewed the PR that announces it. We'll take it. If you want to see the same machinery on your repos, there's a 14-day free trial (no card): codens.ai . Japanese version of the dashboard is here .

Automation is the part of DevOps that actually earns the title. Day 6 was two of the most everyday automation tasks there are, and both had a trap sitting in plain sight. One Linux task, one AWS task. Schedule a recurring job with cron, and launch an EC2 instance from the AWS CLI. The tasks come from the KodeKloud Engineer platform if you want the same lab to work through. Cron: five fields, one rule that trips everyone Cron is the scheduler that has quietly run Linux automation for decades. You hand it a job and a schedule, and its daemon, crond, wakes up every minute to check whether anything is due. If that daemon is not running, nothing fires, so the first move is making sure it is installed and enabled. # Become root sudo su - # Install the cron daemon if it is missing yum install cronie -y # Enable it at boot, start it now, and confirm it is alive systemctl enable crond.service systemctl start crond.service systemctl status crond.service cronie is the package name on RHEL-family distros. enable makes the service survive a reboot, start brings it up right now, and status is how you confirm it is actually running instead of assuming it is. Now edit the crontab: # Edit the current user's crontab crontab -e # minute hour day-of-month month day-of-week command 0 2 * * * /path/to/script.sh # List what is currently scheduled crontab -l That line runs the script every day at 2am. The five fields, in order, are minute, hour, day of month, month, and day of week: minute: 0 to 59 hour: 0 to 23 day of month: 1 to 31 month: 1 to 12 day of week: 0 to 7, where both 0 and 7 mean Sunday Here is the rule that quietly breaks schedules. When you set both the day-of-month field and the day-of-week field to something other than a star, cron treats them as OR, not AND. So 0 0 13 * 5 does not mean midnight on Friday the 13th. It means midnight on every 13th of the month, and also every Friday, whichever lands first. If you actually want the AND, you restrict one field in cron and che

The open-source ecosystem is converging on a hard question: when a commit is written with AI assistance, how do we know — and how do we enforce the disclosure policy? Python's discourse, Linux kernel's Assisted-by trailer, Fedora's AI policy, Apache's disclosure guidelines — every major project is grappling with this. But until now, there has been no tool at the CI level to enforce whatever policy a project chooses. Commit Check v2.11.0 introduces AI Attribution Governance — a new feature that detects known AI tool signatures in commit messages and lets projects decide whether to forbid them outright. To our knowledge, no existing tool enforces this kind of policy at the CI level. The industry need The conversation around AI disclosure is no longer theoretical: The Linux kernel standardized on the Assisted-by: trailer format — but deliberately stopped short of CI enforcement. As Sasha Levin noted at the Maintainers Summit, the kernel sets the convention, not the gate. The Python community is actively discussing whether Claude Code usage should be documented VS Code issue #313962 proposes replacing Co-authored-by with Assisted-by for AI agents Fedora requires AI disclosure (recommends the Assisted-by trailer). QEMU and Gentoo go further and forbid AI-generated contributions entirely. Each community defines its own policy — but none provides a neutral enforcement layer. That is the gap Commit Check fills. Configuration: a single toggle Commit Check keeps it simple. One configuration value, three ways to set it. TOML ( cchk.toml ): [commit] ai_attribution = "forbid" CLI: commit-check --message --ai-attribution = forbid Environment variable: CCHK_AI_ATTRIBUTION = forbid commit-check --message Two modes: Mode Behavior "ignore" No validation (default, backward compatible) "forbid" Rejects any commit containing known AI tool signatures There is no require mode in this release — only ignore and forbid . The reason is pragmatic: requiring an Assisted-by or similar trailer is

Originally published on DevToolHub . Helm 4 shipped in November 2025. Eight months later, most teams are still running Helm 3 in production CI/CD because it works. But Helm 3's final feature release lands September 9, 2026, and security patches stop completely on February 10, 2027. This helm 4 migration is simpler than it looks. Your charts don't need rewriting — Helm 3 Chart API v2 charts are fully compatible with Helm 4. But the automation around Helm has four real breaking points that fail silently if you don't know where to look. [IMAGE: articles/images/2026-07-05-helm-4-migration-guide-featured.png | alt: "helm 4 migration flow from Helm 3 to Helm 4 upgrade path"] Why This Helm 4 Migration Matters Now The EOL timeline has three stages, and they matter differently based on your situation: September 9, 2026 — Final Helm 3 feature release (limited to Kubernetes client library updates only after this date) February 10, 2027 — All security patches stop If your organization runs regulated workloads with requirements around supported software, February 2027 is your hard deadline. But waiting until then means doing this migration under pressure, after 14 months of Helm 4 fixes shipped without you tracking them. The better path: upgrade now, before September, so you're on supported software when new Kubernetes releases land and need updated client libraries. What Actually Broke: The Four Real Changes 1. Post-renderers require plugin registration In Helm 3, you could pass any executable directly to --post-renderer : helm install myapp ./chart --post-renderer ./scripts/mutate.sh Helm 4 drops this. Post-renderers must now be registered as named Helm plugins and referenced by plugin name: helm install myapp ./chart --post-renderer my-post-renderer If your pipeline calls --post-renderer ./path/to/script.sh , it fails on Helm 4. The error message doesn't say "plugin required," so this is easy to miss in a quick smoke test. To wrap an existing script as a plugin, create a plug

When a single engineer manages all the Terraform in an organisation, everything is simple. One repo, one state, one pipeline, one set of credentials. There's no coordination overhead because there's no one to coordinate with. That stops working the moment a second team needs to deploy infrastructure. And by the time you have three or four teams — networking, platform, application, security — the single-team model is actively slowing everyone down. This guide covers what breaks, how teams typically work around it, and how to set up a structure where each team owns their slice of infrastructure independently. What breaks State lock contention Terraform's state locking is per-state. When the networking team is running terraform plan , the application team's pipeline is blocked — even though they're changing completely unrelated resources. The more teams share a state, the more time everyone spends waiting. Blast radius A junior engineer deploying a new application service shouldn't be able to accidentally destroy the VPC. But if application resources and networking resources share a state, a single misconfigured terraform apply can touch anything. Code review catches some of this. Not all of it. Credential sprawl A shared pipeline needs credentials for everything — the networking team's Azure subscription, the application team's AWS account, the security team's DNS provider. Every team's secrets end up in one CI environment, accessible to anyone who can trigger a run. This fails most compliance audits. Approval bottlenecks In many organisations, one person or a small group gatekeeps all infrastructure changes. Every PR needs their review. Every apply needs their approval. The gatekeeper becomes a bottleneck not because they're slow, but because they're a single point of serialisation for all infrastructure work. Backend access as implicit access control Terraform has no built-in concept of per-team or per-workspace permissions. All workspaces in a backend share the sam

Most organisations don't live in a single cloud. You might run compute in AWS, DNS in Cloudflare, identity in Azure AD, and logging in GCP. Terraform handles each provider fine on its own, but the moment you need to coordinate across providers the tooling fights you. This guide walks through the common pain points of multi-cloud Terraform setups and the approaches teams use to cope — then shows how Snap CD makes cross-cloud dependency management a solved problem. Where it gets difficult Credential sprawl Each cloud provider has its own authentication mechanism. AWS uses IAM roles and access keys. Azure uses service principals and managed identities. GCP uses service accounts and workload identity federation. A single Terraform state that spans providers needs credentials for all of them — which means your CI runner or developer workstation holds keys to everything. That's a security problem. A compromised CI pipeline with AWS and Azure credentials exposes both clouds simultaneously. And it's an operational problem — rotating credentials means updating every pipeline that touches that state. This problem compounds at scale: Terraform couples provider processes tightly to credentials , so managing hundreds of accounts across clouds means spawning thousands of provider processes, which quickly becomes unmanageable. Provider version conflicts Terraform providers are versioned independently. Upgrading the AWS provider to fix a bug in aws_eks_cluster shouldn't require you to also test a new version of the Azure provider. But when they share a state, a terraform init -upgrade pulls new versions for everything, and a regression in one provider blocks all deployments. Terraform also lacks built-in support for instantiating multiple providers with a loop and passing providers to modules in for_each , making multi-cloud configurations especially verbose and repetitive. Blast radius across clouds A misconfigured terraform apply in a single-cloud state damages resources in one c

A harness for AI-era Terraform. I'm building one. For a while now I've been developing a harness for infrastructure-as-code as a private SDK and compiler — the layer that sits between whoever proposes a change (a person, an agent, CI) and whatever actually reaches production. This post isn't the tool. It's the thinking underneath it, and the few pieces I've become most convinced by while building it. (Notes from inside the work — where I've landed so far, not advice.) The problem that sent me down this road is easy to state and easy to underrate. A version of it happened recently. An agent fixed some Terraform; the PR read clean — tidy diff, sensible resource names, a plan output that looked exactly like what I'd asked for. It got approved. And then, at apply time, a different plan ran than the one that was reviewed: apply had re-planned against state that moved in between, and the diff that touched production wasn't quite the diff anyone had read. Nothing broke, that time. But that near-miss is the whole reason the harness exists. Because the danger was never "the agent writes bad HCL." Agents write perfectly good HCL; I let them. The danger is the distance between the plan a human reviewed and the plan that actually runs — and once agents are the ones proposing changes at volume, that distance is the thing I most want to nail shut. Where I've landed for now (and expect to keep revising): What AI-era IaC needs isn't AI that can apply . It's a structure where every change — human or agent — is evaluated at the same boundary , and only a reviewed plan ships. The unit of trust isn't the agent. It's a specific, reviewed plan , bound byte for byte. You can't review an agent. You can only review a plan. Instructions to an agent can be broken. A CI gate can't be talked out of it. Put guidance in the prompt; put the guarantee in the gate. Terraform/OpenTofu don't go away. You wrap them in a harness; you don't replace them. Your repo has non-human authors now For years IaC

This is the first article in a four-part series where I document how I turned a 10€/month VPS into a production-grade platform hosting my portfolio, a university group webapplication and a SaaS product, all isolated from each other with Kubernetes. In this part, we take a fresh Ubuntu server and lock it down properly before installing anything else. Why bother with hardening? The moment your VPS gets a public IP address, it starts receiving attacks. Not "might receive", it starts . Within minutes, automated bots will probe port 22, try root:root , admin:admin123 and thousands of other credential combinations. If you skip this step and jump straight to deploying your apps, you are building on sand. The good news: an hour of work is enough to eliminate the vast majority of these threats. Here is what we will set up: A non-root user with sudo privileges SSH key authentication, with passwords and root login disabled UFW as a simple, effective firewall Fail2ban to ban brute-force attackers automatically Automatic security updates What you need A fresh VPS running Ubuntu 24.04 LTS or newer. I use a Hostinger KVM 2 (2 vCPU, 8 GB RAM, 100 GB NVMe), but any provider works: Hetzner, DigitalOcean, OVH, Contabo. The root password or SSH key your provider gave you. A terminal on your local machine (macOS, Linux, or WSL on Windows). Throughout this tutorial, replace YOUR_SERVER_IP with your server's IP address and deploy with the username you want to use. Step 1: First login and system update Connect as root for the first and last time: ssh root@YOUR_SERVER_IP Update everything before touching anything else: apt update && apt upgrade -y apt autoremove -y If a kernel update was installed, reboot now: reboot Wait a minute, then reconnect. Step 2: Create a non-root user Working as root is like driving without a seatbelt: fine until it isn't. One mistyped rm -rf and the party is over. Create a dedicated user: adduser deploy Choose a strong password (you will still need it for sudo ,