标签：#cicd

Six failed CI runs. Thirty-six hours of GitHub Actions time. Every run timing out at exactly the 6-hour limit. The culprit was one line in tests/setup.js . The Setup We were building a multi-tenant platform with a PostgreSQL backend — around 76 database models handling everything from user accounts and billing to visitor logs and real-time notifications. The test suite had grown to roughly 1,140 test cases across 36 files. Standard stuff. CI ran on every PR. Tests passed locally. And then one day, CI just... never finished. The Anti-Pattern Here's what the test setup looked like: // tests/setup.js beforeEach ( async () => { const tableNames = await getTableNames (); // 76 tables await sequelize . query ( `TRUNCATE TABLE ${ tableNames . join ( ' , ' )} CASCADE;` ); }); The intent was clean isolation — every test starts with a blank slate. Reasonable in theory. Catastrophic in practice. The Math Do the multiplication: 76 tables × 1,140 tests = 86,640 TRUNCATE operations Each TRUNCATE TABLE ... CASCADE is not a cheap operation. PostgreSQL has to: Acquire exclusive locks on all referenced tables Walk the foreign key graph to find dependent tables Truncate each in dependency order Release locks With a moderately complex schema where most tables reference others (users → societies → members → invoices → payments → ...), a single TRUNCATE ... CASCADE on a central table can fan out into dozens of implicit truncations. Multiply that by 86,640 and you have a test suite that will never complete within any reasonable timeout. Why It Wasn't Caught Sooner Two reasons: 1. It used to be fast. When the suite had 50 tests and 20 tables, this pattern worked fine. 50 × 20 = 1,000 truncations — uncomfortable but survivable. Nobody noticed when the suite crossed a tipping point. 2. Local runs used a different database state. Locally, developers often ran a subset of tests with --grep or file-specific runs. The full suite was only ever run on CI, and CI was slow enough that most assumed i

Most teams treat "monorepo vs multi-repo" as a binary choice. Pick one, commit, move on. We ended up with a hybrid, and it turned out to be the right call — not out of indecision, but because our apps have genuinely different deployment and ownership characteristics. Here's what we built, why, and what it costs. The System The platform consists of six applications: App Type Primary Users REST API backend Node.js + TypeScript — (consumed by all apps) Society dashboard React web app Society managers, admins, accountants Company admin panel React web app Internal operations Marketing website Next.js Public Resident mobile app React Native (Expo) Residents Guard mobile app React Native (Expo) Security personnel All six apps talk to the same API. But they have very different deployment cycles, team ownership, and testing requirements. The Structure: 4 Repositories repo: main-platform (monorepo) ├── api/ — Express + Prisma backend ├── web-society/ — Society dashboard ├── web-admin/ — Company admin panel └── web-marketing/ — Marketing site repo: mobile-resident — Resident app (React Native) repo: mobile-guard — Guard app (React Native) repo: mobile-staff — Society staff mobile app (React Native) The web apps and the API live together in one monorepo. The three mobile apps each have their own repository. Why Split Mobile From Web? The driving factor was deployment cadence and review process . Web apps deploy on push — merge to main, CI builds, CDN updated within minutes. The feedback loop is fast, rollbacks are instant, and there's no approval gate between code and production. Mobile apps go through app store review. A release cycle includes building a release APK, submitting to Google Play (and Apple App Store), waiting for review, and then a staged rollout. The cadence is measured in days, not minutes. Mistakes are expensive to reverse — a bad release means submitting a patch, waiting again, and potentially having a broken version live for days. Given that difference, mob

One of the biggest mistakes beginners make when learning Terraform is treating their local machine as the deployment server. A typical workflow looks like this: terraform init terraform plan terraform apply While this approach is perfectly fine for learning, it quickly becomes problematic when working on real-world projects with multiple engineers. Consider these questions: Who deployed the infrastructure? Was the infrastructure reviewed before deployment? Can someone else reproduce the deployment? What happens if the engineer's laptop is lost or misconfigured? How do we know exactly what changed? These are some of the reasons Infrastructure as Code (IaC) is almost always integrated with Continuous Integration and Continuous Deployment (CI/CD) pipelines in professional environments. In this article, we'll build a simple Terraform CI/CD pipeline using GitHub Actions. Instead of focusing only on the YAML syntax, we'll first understand why each stage exists and how they work together to produce safe, repeatable infrastructure deployments. What is Terraform CI/CD? Terraform CI/CD is the process of automating the validation, planning, and deployment of infrastructure whenever changes are made to Terraform code. Instead of running Terraform commands manually from a developer's laptop, a CI/CD platform executes those commands automatically in a controlled environment. The workflow typically looks like this: Developer │ ▼ Git Push │ ▼ GitHub Repository │ ▼ GitHub Actions │ ▼ Terraform Init │ ▼ Terraform Validate │ ▼ Terraform Plan │ ▼ Manual Approval │ ▼ Terraform Apply │ ▼ AWS Infrastructure This approach provides consistency, visibility, and security while reducing the chances of human error. Why Not Run Terraform Manually? Running Terraform from your laptop works well for personal projects, but it introduces several risks in a team environment. Manual Deployment CI/CD Deployment Requires someone to remember every command Runs automatically Easy to skip validation Validat

Over the past few months, I've been exploring how AI agents can fit into a modern CI/CD pipeline—not to replace engineers, but to eliminate repetitive work that slows teams down. Here's what worked well: ✅ Automatically categorized incoming pull requests ✅ Flagged potential security and dependency issues ✅ Suggested fixes for linting and test failures ✅ Generated review summaries for faster code reviews ✅ Reduced context switching for reviewers The biggest lesson? AI is most valuable before the human review begins. The Problem Every engineering team eventually runs into the same issue. Developers submit pull requests faster than reviewers can process them. A typical PR often goes through several repetitive steps: CI builds Unit tests Linting Dependency checks Security scanning Style comments Reviewer assignment Documentation validation None of these tasks require deep architectural thinking, yet they consume valuable engineering time. I started wondering: What if an AI agent handled the first round of triage automatically? The Workflow Instead of waiting for a human reviewer, the pipeline lets an AI agent inspect every pull request immediately after CI starts. Developer │ ▼ Pull Request Created │ ▼ CI Pipeline Starts │ ▼ AI Agent ├── Analyze changed files ├── Review commit summary ├── Detect risky changes ├── Check coding standards ├── Explain failing tests ├── Suggest fixes └── Generate PR summary │ ▼ Human Review By the time a reviewer opens the PR, much of the routine analysis is already complete. Example GitHub Actions Workflow A simplified workflow might look like this: name: AI Pull Request Review on: pull_request: types: [opened, synchronize] jobs: ai-review: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run Tests run: npm test - name: Run Linter run: npm run lint - name: AI PR Analysis run: ./scripts/ai-review.sh The AI step can analyze: Test failures Lint violations Changed files Security findings Dependency updates before publishing a r

A practical framework for test planning, hardware scheduling, artifact traceability, failure classification, and evidence-based quality gates Disclaimer: The views expressed in this article are my own. The architecture, examples, terminology, and code snippets are generalized for educational purposes and do not describe or disclose any employer’s proprietary systems, confidential information, or internal implementation details. A software change can compile successfully, pass unit tests, and still introduce a serious GPU regression. The failure may appear only on one GPU generation. It may depend on a particular driver, firmware revision, operating system, graphics API, or workload. A change may preserve functional correctness while quietly reducing performance. It may also cause an intermittent failure that disappears when the test is rerun. This is why GPU validation cannot be treated as conventional CI/CD with a GPU runner attached to the end of the pipeline. A dependable GPU validation platform must coordinate: Software and firmware artifacts Hardware configurations Test coverage GPU resource scheduling Failure classification Performance baselines Engineering evidence It must do all of this while operating under an important constraint: compatible GPU capacity is limited and expensive. The objective is not simply to run more tests. It is to produce reliable evidence quickly enough to support engineering decisions. Why conventional CI/CD is not enough A conventional application pipeline often resembles: Commit ↓ Build ↓ Unit tests ↓ Integration tests ↓ Deployment A GPU validation pipeline is more multidimensional: Code or configuration change ↓ Build software and firmware artifacts ↓ Determine affected GPU configurations ↓ Reserve compatible hardware ↓ Prepare the driver and runtime environment ↓ Run functional, stability, and performance tests ↓ Collect logs, traces, metrics, and crash artifacts ↓ Classify failures and compare results with baselines ↓ Make a mer

Originally published on hexisteme notes . I have a small pipeline that crossposts my notes to dev.to. It parses a Markdown file's front matter, builds a payload, and calls the dev.to API to publish. It has 19 gate tests, and every one of them was green the whole time it was shipping. It published nine articles. All nine went live with their titles broken — the front-matter quotes were sitting right there in the title, visible to anyone who looked, for nine days, and nothing in the pipeline noticed. I didn't notice either. A human had to open the dev.to profile page by accident before anyone found out. This is the postmortem, and the reason I'm writing it up as a general essay rather than just a fixed-bug log is that the root cause isn't specific to dev.to, or to Markdown front matter, or to Python. It's a category of mistake that any pipeline with an external endpoint on the other end can make: testing the payload you build, and never testing what the other system does with it. The pipeline that had "passed everything" The shape of it is ordinary. A draft file has YAML-style front matter — title: "Some Title" — because that's the convention. A parser reads the front matter and pulls out the title. A payload builder takes that title and a few other fields and assembles the JSON body for the dev.to API. The API gets called, dev.to accepts it, the article is live. Nineteen gate tests cover this path — the front-matter parsing and the payload/API contract of the pipeline's own code. All green, every publish. The gap is in what "parses the front matter" actually means. The parser isn't a real YAML parser. It's closer to line.partition(":") — split each line on the first colon, take the right-hand side as the value. That works fine for tags: testing, devops where there's nothing to unwrap. It does not work for title: "Some Title" , because the quote characters are part of the string on the right-hand side of the colon, and a partition-based parser has no concept of "this

The moment we realized our staging environment was broken It was 3 PM on a Thursday, and our team was scrambling. A critical API change had just been merged to main, but the staging environment—our supposed safety net—was showing false positives. The integration tests passed, but the mobile app was completely broken in production. That's when we knew: our shared staging environment was failing us. The Problem: Shared Staging Is Broken by Design Like many engineering teams, we operated with a single, shared staging environment. Every developer deployed their changes to the same place, leading to: Deployment conflicts: "Who deployed that breaking change?" Cascading failures: One broken PR would block the entire team Test contamination: Data from one test would leak into another Delayed feedback: You'd only discover issues after merging your PR and deploying to staging The "works on my machine" syndrome, now at scale The worst part? Our API contracts were changing constantly, but we only discovered breaking changes during integration testing—often too late. The Solution: Ephemeral Environments per PR We made a radical change: every PR gets its own isolated, short-lived environment. Here's our architecture: Our Implementation Stack Infrastructure: Kubernetes (EKS) with namespace-per-PR Orchestration: Custom GitHub Action workflow Database: Isolated RDS instance per environment Contract Testing: Pact flow + OpenAPI validation Cleanup: AWS Lambda that runs every hour, destroying environments older than 2 hours The Game Changer: Automated Contract Testing The magic wasn't just in isolated environments—it was in what we did with them. Every time a PR deployed to its ephemeral environment, we ran: Consumer-Driven Contract Testing (Pact) Our mobile and web clients would verify their expectations against the actual deployed API. If a change broke what the client expected, the PR would fail. Provider Contract Validation We'd automatically verify that the deployed API matched ou

* When a project first starts, CI/CD pipelines are usually simple. * Build the application. Run a few tests. Deploy somewhere. Done. Then six months pass. Another test gets added. Then another deployment step. A security scan. Performance tests. Notifications. More environments. Before long, a pipeline that once took five minutes now takes forty-five. I've seen this happen more than once, and it's rarely because Azure DevOps is the problem. It's usually because nobody ever stops to ask one simple question: Does this step still belong here? Everything Ends Up in the Same Pipeline One of the most common mistakes I see is trying to make a single pipeline do everything. Every Pull Request ends up running: Every unit test Every API test Hundreds of UI tests Security scans Deployment steps Report generation The result? Developers wait longer for feedback, releases become slower, and people eventually start ignoring failed pipelines because they happen too often. Fast Feedback Wins Not every test needs to run on every commit. A better approach is to think about the purpose of each pipeline. For a Pull Request, I want answers quickly. That usually means: Build the application Run unit tests Run a small smoke test suite Stop if something important fails Everything else can happen later. Long-running regression tests, cross-browser testing and other expensive checks are often better suited to scheduled or nightly pipelines. Pipelines Should Evolve A pipeline isn't something you build once and forget about. Every few months it's worth reviewing it. Ask yourself: Which step takes the longest? Which tests fail most often? Are there any tasks nobody remembers adding? Are we getting useful feedback, or just more output? Removing unnecessary work is just as valuable as adding new automation. Final Thoughts Azure DevOps is an incredibly powerful platform, but even the best tools become frustrating if they're overloaded with unnecessary work. The goal isn't to build the biggest pipel

One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever. Six months later that same framework was a mess. Three different judge models with three different parsing hacks. A test dataset nobody had touched since November. A CI gate that kept failing because a vendor nudged their model, not because anyone broke a prompt. And the second engineer on rotation asking me, fairly, "how does this even work?" The framework did not fail. The eighty percent of the work the weekend tutorial skipped is what failed. That gap is the whole story, and this is what I would tell myself before starting again. The one line I wish someone had told me: build the rubric, buy the runner Here is the split that took me six months to see. Some parts of an eval setup are yours and only yours. The rubric that decides what "good" means for your product. The dataset built from your real failures. The rules for when a change is bad enough to block a release. Nobody else can write these, because they encode your domain. The rest is the same at every company. The thing that calls the judge model, parses its answer, retries, and caches. The machinery to run thousands of checks in parallel. The plumbing that scores live traffic. The system that groups failing calls together. Every team rebuilds these, hits the same bugs, and gains nothing by writing them twice. So build the first list. Do not hand-build the second. I rebuilt the second, and it cost me most of a year. Two questions I now ask about every single piece Before writing any part of this, I ask two things: Is it specific to me, or generic? A rubric for my domain is specific and worth owning. A retry-and-cache loop around a model call is generic. Everyone writes the same one. Does it compound, or does it rot? A dataset that grows from real production failures compounds. A year in, it is a regression suite no competitor can copy. A hand-built tracing layer rots. The moment a vendor chan

Browser automation used to be easier to describe. A test opened a page, filled in a form, clicked a button, and checked the result. The hardest parts were usually selectors, waits, and browser compatibility. Those problems still exist, but the surface area has expanded. Today, browser tests may need to handle streaming interfaces, MFA, AI-generated content, multiple operating systems, preview deployments, canary releases, and code changes proposed by AI assistants. The challenge is no longer just writing a script that passes. The challenge is building a testing system that remains understandable and affordable after hundreds of tests and thousands of CI runs. Start by measuring instability instead of normalizing it Flaky tests often become accepted background noise. A test fails, CI retries it, and the second run passes. The pipeline turns green, so the team moves on. Over time, the retry count grows and nobody is sure which failures matter. The problem is that a passing retry does not erase the cost of the first failure. The article on calculating the real cost of flaky test retries in CI provides a useful framework for evaluating compute costs, developer interruptions, delayed feedback, and investigation time. A simple reliability metric can help: first-attempt pass rate = tests passing without retry / total test executions This is often more revealing than the final pipeline pass rate. A suite with a 99% final pass rate may still be deeply unstable if many tests require multiple attempts. Reproduce the environment before changing the test When a browser test fails only in CI, teams often edit the test before reproducing the environment. That can lead to unnecessary waits and conditionals. One of the most common variations is a test that passes in visible Chrome but fails in headless mode. The explanation is not always “headless Chrome is flaky.” Differences in viewport, rendering, animation, fonts, and resource timing can all change application behavior. This det

I’m currently relearning the Core Three — HTML, CSS, and JavaScript — as I work toward becoming a full-stack JavaScript developer. Before I came back to learning software, I spent 22 years working industrial turnarounds. One lesson from that world has followed me into software engineering: Never trust a single point of failure. In industrial maintenance, there’s a safety practice called double block-and-bleed . Instead of trusting one isolation valve, you use two independent valves with a bleed point between them. If one valve leaks, you know immediately. The entire system assumes individual components can fail. Safety doesn’t come from perfect parts. It comes from independent layers of protection. That idea completely changed how I think about CI pipelines. When I first started relearning web development, my mindset was simple: Run Lighthouse. Everything green? Great. 100 across the board locally? Even better. Ship it. Different results after deployment? Uh-oh. Now I see Lighthouse as one checkpoint — not the finish line. A fast website can still have accessibility issues. An accessible site can still have broken metadata. Good SEO won’t catch rendering bugs. Passing unit tests won’t tell you if the generated HTML is malformed. Every tool has blind spots. No single tool should get the final vote. So instead of asking: “Did my tests pass?” I ask: “What kinds of failures could still slip through?” That question naturally leads to layered validation. Formatting Linting Type checking Accessibility checks Performance audits HTML validation SEO analysis Manual review None of these tools is perfect. Together, they’re much stronger than any one of them alone. The more I learn about software, the more I find myself applying lessons from heavy industry. Different environment. Different risks. The same engineering mindset. Assume components will fail. Design systems that fail safely. That’s becoming the philosophy behind every test matrix and CI pipeline I’m designing. What’s

Bạn có thể chạy bài kiểm tra API của Apidog CLI trong Drone CI bằng một bước Docker dùng ảnh Node, cài apidog-cli , rồi gọi apidog run với test scenario và environment tương ứng. Token Apidog nên được lưu trong Drone Secret và inject vào pipeline bằng from_secret . Bài viết này cung cấp cấu hình .drone.yml có thể copy-paste, cách quản lý secret, giới hạn chạy theo branch/event và cách xuất báo cáo khi Drone không có artifact storage tích hợp sẵn. Dùng thử Apidog ngay hôm nay Drone CI là gì và hoạt động như thế nào Drone là nền tảng CI/CD mã nguồn mở, chạy theo mô hình container-native và hiện là một phần của Harness. CI/CD là thực hành tự động build, test và phân phối phần mềm trên mỗi thay đổi mã nguồn. Nếu cần ôn lại nền tảng, xem thêm CI/CD là gì . Điểm quan trọng của Drone: mỗi step trong pipeline chạy trong một Docker container riêng. Bạn chọn image cho từng step, sau đó Drone chạy các command trong container đó. Cách này giúp pipeline dễ tái tạo, dễ debug và không phụ thuộc vào một build agent đã cài sẵn nhiều công cụ. Pipeline được khai báo trong file .drone.yml ở thư mục gốc repository. Một Docker pipeline thường có: kind type name steps Ví dụ tối thiểu: kind : pipeline type : docker name : api-tests steps : - name : greeting image : alpine commands : - echo hello - echo world Drone chạy commands như shell script với cơ chế fail-fast. Nếu một command trả về exit code khác 0 , build sẽ fail. Working directory mặc định là thư mục gốc của repository. Vì sao nên chạy API test trong container step API test giúp phát hiện sớm các thay đổi phá vỡ contract, ví dụ: response schema thay đổi ngoài ý muốn status code khác kỳ vọng field bắt buộc bị thiếu logic xác thực hoặc phân quyền bị lỗi Với Apidog, bạn thiết kế và duy trì test scenario trong UI, sau đó chạy lại chính các scenario đó từ CLI. CI không cần giữ thêm collection file riêng hoặc viết lại test script. Để xem thêm về cách đưa API testing vào pipeline, đọc các phương pháp hay nhất về CI/CD cho kiểm tra API .

Test Isolation: A Lesson I Learned While Migrating Playwright Tests During my software engineering internship, I helped optimize our CI pipeline by identifying which E2E tests could safely run in parallel. That work quickly taught me that the biggest obstacle wasn't Playwright or Python, it was test isolation. This article is about that lesson. What is test isolation? A simple rule I now use is this: if a test can't run by itself with the same outcome, it probably isn't truly isolated. A well-isolated test should produce the same result whether it: runs by itself runs first or last runs after another test runs in parallel with hundreds of other tests To understand test isolation, it also helps to understand what state means. State isn't limited to database rows. During the migration, I found tests interacting with many different kinds of state. database records global configuration filesystem resources application caches If any of these are shared between tests, they become potential sources of hidden dependencies. How tests lose isolation As I started reading the existing test suite, I noticed a recurring pattern. Many tests assumed something about the environment instead of creating it themselves. Some expected specific data to already exist. Others modified global settings without restoring them afterward. Some searched for rows based on their position in a table instead of using a stable identifier like a name or ID. None of these looked particularly problematic when reading a single test. The problems only appeared once the entire suite started running together. One test would leave behind data another test didn't expect. A shared configuration would silently affect unrelated tests. A UI assertion would suddenly fail because another test inserted an extra row into the same table. Individually, the tests appeared independent. Together, they formed hidden dependencies. Not all shared state is equally difficult to isolate One realization that helped me reason abou

When a single engineer manages all the Terraform in an organisation, everything is simple. One repo, one state, one pipeline, one set of credentials. There's no coordination overhead because there's no one to coordinate with. That stops working the moment a second team needs to deploy infrastructure. And by the time you have three or four teams — networking, platform, application, security — the single-team model is actively slowing everyone down. This guide covers what breaks, how teams typically work around it, and how to set up a structure where each team owns their slice of infrastructure independently. What breaks State lock contention Terraform's state locking is per-state. When the networking team is running terraform plan , the application team's pipeline is blocked — even though they're changing completely unrelated resources. The more teams share a state, the more time everyone spends waiting. Blast radius A junior engineer deploying a new application service shouldn't be able to accidentally destroy the VPC. But if application resources and networking resources share a state, a single misconfigured terraform apply can touch anything. Code review catches some of this. Not all of it. Credential sprawl A shared pipeline needs credentials for everything — the networking team's Azure subscription, the application team's AWS account, the security team's DNS provider. Every team's secrets end up in one CI environment, accessible to anyone who can trigger a run. This fails most compliance audits. Approval bottlenecks In many organisations, one person or a small group gatekeeps all infrastructure changes. Every PR needs their review. Every apply needs their approval. The gatekeeper becomes a bottleneck not because they're slow, but because they're a single point of serialisation for all infrastructure work. Backend access as implicit access control Terraform has no built-in concept of per-team or per-workspace permissions. All workspaces in a backend share the sam

Most organisations don't live in a single cloud. You might run compute in AWS, DNS in Cloudflare, identity in Azure AD, and logging in GCP. Terraform handles each provider fine on its own, but the moment you need to coordinate across providers the tooling fights you. This guide walks through the common pain points of multi-cloud Terraform setups and the approaches teams use to cope — then shows how Snap CD makes cross-cloud dependency management a solved problem. Where it gets difficult Credential sprawl Each cloud provider has its own authentication mechanism. AWS uses IAM roles and access keys. Azure uses service principals and managed identities. GCP uses service accounts and workload identity federation. A single Terraform state that spans providers needs credentials for all of them — which means your CI runner or developer workstation holds keys to everything. That's a security problem. A compromised CI pipeline with AWS and Azure credentials exposes both clouds simultaneously. And it's an operational problem — rotating credentials means updating every pipeline that touches that state. This problem compounds at scale: Terraform couples provider processes tightly to credentials , so managing hundreds of accounts across clouds means spawning thousands of provider processes, which quickly becomes unmanageable. Provider version conflicts Terraform providers are versioned independently. Upgrading the AWS provider to fix a bug in aws_eks_cluster shouldn't require you to also test a new version of the Azure provider. But when they share a state, a terraform init -upgrade pulls new versions for everything, and a regression in one provider blocks all deployments. Terraform also lacks built-in support for instantiating multiple providers with a loop and passing providers to modules in for_each , making multi-cloud configurations especially verbose and repetitive. Blast radius across clouds A misconfigured terraform apply in a single-cloud state damages resources in one c

In modern software development, Continuous Integration and Continuous Deployment (CI/CD) pipelines...

Use Docker and a disposable mailbox to verify AWS SES delivery in CI before release without touching shared inboxes or leaking test mail. Shipping an app that "sent the email" is not the same as shipping an app that delivered a usable message. In AWS stacks, the miss usually shows up late: a container points at the wrong SES region, a preview environment uses stale credentials, or the message lands with broken links after templating. A disposable email address check is a simple final guardrail, and its easy to automate when your mail sender already runs in Docker. This is the pattern I recommend when teams are searching for temp mail com style testing, but need something cleaner for production-like CI. The goal is not to replace unit tests or the Amazon SES mailbox simulator . The goal is to prove that your real app container, with real environment wiring, can send one controlled message and that the inbox content is worth shipping. Why this check belongs in the release path AWS SES is strict in useful ways. If your account is still in the sandbox, you can only send to verified identities or the mailbox simulator . Even after production access, delivery problems still happen becuase the application layer and the cloud layer drift separately. A practical smoke test catches things that mocks often miss: wrong AWS_REGION or SES endpoint selection missing secrets in the container runtime template variables that render empty in preview builds broken confirmation links caused by bad environment URLs unexpected sender identity changes between staging and release That list looks basic, but it doesnt stay basic when three services, one worker, and a release job all touch outbound email. A small Docker pattern that keeps the test deterministic Keep the sender in the same Docker image you deploy, but trigger a narrow email scenario from CI. For example, seed a temporary user, call the notification path once, then poll a disposable inbox API for the matching subject line. servi

I once spent forty minutes at eleven at night debugging a deploy that wasn't broken. The release script ran the database migration, the migration threw connection refused , the script exited non-zero, the deploy rolled itself back, and I got paged. So I did the things you do. I read the migration. I read the logs. I checked the database — it was up, it was healthy, it accepted my connection instantly. I re-ran the deploy and it worked. I chalked it up to gremlins and went to bed, which is the part I'm not proud of, because it happened again two days later. That time I watched the timing: the script brought up a fresh database container and started the migration about six seconds before Postgres finished initializing and began accepting connections. The migration was racing the database's boot. Most of the time it won. The times it lost, I lost forty minutes. The script wasn't wrong about anything except one assumption: that a dependency is ready the instant you ask for it. In production, dependencies are eventually ready That's the mental model shift. Networks blip. A service you call returns a 503 for the two seconds it takes to finish a rolling restart. An API rate-limits you with a 429 it fully expects you to retry. A fresh container's database isn't accepting connections for its first few seconds. Treating the first failure as fatal turns every one of these normal, transient conditions into a paged engineer — and the script that handles them isn't smarter — it declines to give up on the first try. But retrying naively is its own trap. Retry instantly and you hammer a recovering service into staying down. Retry forever and a genuinely dead dependency hangs your script indefinitely. Retry a 404 and you wait a minute to confirm what you already knew. Good retries are bounded, backed off, and selective. A retry function you can reuse anywhere #!/bin/bash # Purpose: survive transient failures instead of dying on the first error set -euo pipefail CHECK = "✓" CROSS = "

The Quest Begins (The "Why") Honestly, I was tired of playing “guess the state” every time I spun up a new environment. One day I clicked “Apply” in the AWS console, watched a handful of EC2 instances, S3 buckets, and IAM roles appear, and then realized I had no idea how to recreate that exact setup six months later when the team needed a staging copy. It felt like trying to rebuild the Death Star from memory after a single glance at the blueprints—frustrating, error‑prone, and definitely not the heroic saga I signed up for. That moment was my “aha!”: I needed a repeatable, version‑controlled way to describe infrastructure. Enter Infrastructure as Code (IaC). I’d heard the buzz, but the real question was which tool to wield—Terraform or CloudFormation? Both promised declarative provisioning, but they spoke different dialects. I decided to embark on a quest to learn both, slay the configuration drift dragon, and come out with a reusable spellbook I could share with anyone on the team. The Revelation (The Insight) The breakthrough came when I stopped thinking of IaC as “just another config file” and started seeing it as a storytelling language . Every resource block is a character, every variable a plot twist, and the state file the ever‑growing script that remembers what happened in previous chapters. When I wrote my first Terraform module, it felt like Neo realizing he could bend the spoon—suddenly the impossible became trivial. I could define a VPC, subnets, security groups, and an RDS instance in a few dozen lines, run terraform init , terraform plan , and watch the plan show exactly what would change before any resources touched the cloud. No more surprise “you created a public‑facing DB!” moments. CloudFormation, on the other hand, felt like the loyal sidekick that already lives in the AWS universe. Its JSON/YAML templates are native to AWS, so there’s no extra provider to install, and drift detection is built‑in. The trade‑off? A bit more verbosity and a steepe

If your Terraform plans are slow, your blast radius is too wide, or multiple teams are stepping on each other's changes, it's time to split your monolith. See The Problem with Large Terraform States for how to diagnose whether you've reached that point. This guide walks through the process of breaking a monolithic Terraform state into smaller, focused states — and how Snap CD can manage the dependencies between them so you don't have to. The approach 1. Identify natural boundaries Look at your resources and group them by lifecycle and ownership. Common boundaries: Networking — VPCs, subnets, route tables, NAT gateways. Changes rarely, underpins everything. DNS — Zones, records. Usually owned by a platform team. Compute — Kubernetes clusters, VM scale sets, container services. Changes more often, depends on networking. Application infrastructure — Databases, caches, queues, storage accounts. Owned by application teams. Monitoring — Dashboards, alerts, log sinks. Changes frequently, depends on everything but nothing depends on it. A useful test: if two resources would never be changed in the same PR by the same person, they probably belong in different states. 2. Map the dependencies Before you move anything, draw the dependency graph. Which groups produce values that other groups consume? networking dns │ ▲ ▼ │ compute ──────────►─┘ │ ▼ application │ ▼ monitoring The outputs that cross these boundaries are what you'll need to wire up after the split. Typical examples: Networking → Compute: vpc_id , private_subnet_ids Compute → DNS: load_balancer_ip Compute → Application: cluster_endpoint , cluster_ca_certificate Application → Monitoring: database_id , cache_name 3. Use terraform state mv to migrate resources Terraform's state mv command lets you move resources from one state to another without destroying and recreating them. # Initialize the destination state cd modules/networking terraform init # Move resources from the monolith to the new state terraform state mv \