标签：#cicd

In Modern Time applications are no longer simple monolithic systems. Today organizations run: Microservices Kubernetes Containers Serverless Functions Multi-Cloud Platforms Distributed Systems As infrastructure becomes more distributed, troubleshooting becomes significantly harder. A single user request may travel through: Frontend ↓ API Gateway ↓ Microservice A ↓ Microservice B ↓ Database When something breaks, the biggest challenge becomes: "What exactly happened?" This is where Observability becomes critical. 🔗 Resources ** Support the Journey on GitHub: If you're following along, consider starring and forking the repo:** https://github.com/17J/30-Days-Cloud-DevSecOps-Journey What is Observability? Observability is the ability to understand the internal state of a system by analyzing the data it produces. In simple words: Can we understand what is happening inside our systems? Observability helps engineers answer: Why is the application slow? Which service is failing? Which request caused the issue? What changed recently? Where is latency occurring? Without observability: Problem Exists ↓ Guessing Begins With observability: Problem Exists ↓ Evidence Available ↓ Faster Resolution Why Observability Matters Modern cloud-native systems generate enormous amounts of data. Example: 100 Microservices ↓ Millions of Requests ↓ Thousands of Containers Traditional monitoring alone is no longer sufficient. Organizations need: Visibility Insights Correlation Root Cause Analysis Observability provides all of them. Monitoring vs Observability Many people confuse monitoring and observability. Monitoring asks: What is wrong? Observability asks: Why is it wrong? Example: Monitoring: CPU Usage = 95% Observability: Which service? Which request? Which dependency? Which deployment caused it? Observability provides context. The Three Pillars of Observability Modern observability is built on three primary pillars. Metrics Logs Traces Or: Monitoring Logging Tracing Together they provide a

7 Infra Improvement Strategies to Prevent Next.js Deployment Build Failures in 2026 Recently, our team's deployment pipeline started showing serious instability. Specifically, we encountered recurring build failures related to the chat build. As a result, the entire development team was preoccupied with battling these build failures. Attempts and Pitfalls Initially, I thought the --preload detection logic was the problem. I modified it to detect only specific lines, but this ended up causing issues in other areas. The recurring chat build failures were actually caused by the next.config file not properly recognizing file extensions. I modified it to allow extensions like .mjs , .js , .ts , and .cjs , but even that didn't work correctly at first, leading to some wasted effort. # .github/workflows/deploy.yml (Excerpt from initial version) - name : Run Preload Detection run : | # ... existing logic ... if [[ "$LINE" == *"some_pattern"* ]]; then echo "Preload detected" # ... fi I modified it to detect only specific lines like the above, which led to unintended behavior. // next.config.js (Initial configuration) module . exports = { // ... experimental : { // ... }, // ... }; Regarding extensions, I initially allowed only a few types, and only after experiencing chat build failures did I modify it to support more extensions. Root Causes In the end, it was a combination of several complex issues. There were flaws in the --preload detection logic, and the range of supported extensions in the next.config file was too narrow, which was the direct cause of the chat build failures. Additionally, there was confusion arising from the chat server builds being inconsistent between P1/P2 and P0 stages. Problems also occurred because the .next directory was not preserved, and the smoke gate was too lenient, failing to catch build failures. Finally, there was an unexpected side effect where the next/font/google library caused GCE outbound connection errors. Solutions To address these

Parent/child first. Evidence emission second. Ansible control plane third. Every release was a manual evidence collection exercise. The pipeline was the bottleneck. This is a redacted write-up of a real engagement: rebuilding a healthcare SaaS company's CI/CD pipeline across a fleet of Linux hosts on AWS. The context The engineering team had grown faster than the pipeline architecture had evolved. What started as a single-stage GitLab job for a small team had been extended, patched, and worked around as the team scaled past the patterns the original pipeline was built for. The result was familiar. Each deploy took 30 to 45 minutes of mostly-serial execution. Engineers had developed informal habits to work around the slowness, including pushing partial changes outside the pipeline when the timeline got tight. Audit windows were preceded by three-week sprints in which the team manually compiled deployment logs, screenshots of access reviews, and approval chains into PDFs describing what the pipeline was supposed to be doing. The work was technically passing HIPAA audits, but the audit was a snapshot of a system the auditor could not independently verify. The cost was paid twice: the velocity loss on every deploy, and the three-week scramble before each assessment. The team knew the architecture was wrong. They needed engineering hands to redesign it without slowing the product roadmap the audits were already eating into. The approach The redesign moved in three layers. First, decompose the monolithic pipeline into parent/child stages so work can parallelize and the audit boundary of each stage is provable. Second, build structured evidence emission into every stage as a property of how it runs, not an after-the-fact compilation task. Third, layer an Ansible control plane across the host fleet so HIPAA control state is continuously validated, not reviewed quarterly. ┌────────────────────────── Parent Pipeline (.gitlab-ci.yml) ──────────────────────────┐ │ │ │ ┌────────

The first time I deployed a Next.js app to production, it took me three days. Not because the app was complicated — it was a straightforward portfolio site. It took three days because I had no idea what I was doing with AWS, I'd never written a GitHub Actions workflow, and every tutorial I found either skipped the hard parts or assumed I already knew them. By the time I was done, I had a deployment pipeline I was genuinely proud of: push to main, GitHub Actions runs the build, tests pass, the app deploys to an EC2 instance behind CloudFront. Zero manual steps. Zero downtime deploys. Total cost: about $5/month. This guide is the one I wish had existed. We're going to deploy a Next.js app to AWS from scratch — EC2 for compute, CloudFront for CDN, GitHub Actions for CI/CD — with every step explained so you understand what you're building, not just copying commands. Why AWS Instead of Vercel? This is a fair question. Vercel is genuinely excellent for Next.js, and for most projects it's the right call. You push, it deploys. Done. AWS makes sense when: You need to control the infrastructure (compliance, data residency, custom VPC configuration) You're running other services (databases, queues, lambdas) in AWS and want everything in the same network You want to learn infrastructure skills that transfer to enterprise environments Your app has specific performance requirements that benefit from custom CloudFront configuration You're a freelancer or consultant who wants to bill separately for infrastructure If none of those apply to you, use Vercel. This guide is for when they do. The Architecture Here's what we're building: ┌────────────────────────────────────────────────────────┐ │ GITHUB ACTIONS CI/CD │ │ │ │ Push to main → Build → Test → Deploy to EC2 │ └──────────────────────┬─────────────────────────────────┘ │ SSH deploy ▼ ┌────────────────────────────────────────────────────────┐ │ AWS EC2 INSTANCE │ │ │ │ Ubuntu 22.04 LTS │ │ Node.js 20 + PM2 (process manager) │ │ N

Appendix: Live System Output — Real Pipeline in Production All output below was captured live from the running pipeline on 2026-03-08. These are not mock outputs — they come from actual AWS infrastructure and Kubernetes clusters. ArgoCD — All 50 Applications Across 6 Clusters The following is the live output of argocd app list from the hub cluster ( myapp-production-use1 ). Every component of the pipeline is represented — security, logging, monitoring, backups, and the application itself. $ argocd app list --output wide NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH argocd/argo-rollouts-myapp-production-use1 myapp-production-use1 argo-rollouts production Synced Healthy argocd/argo-rollouts-myapp-production-usw2 myapp-production-usw2 argo-rollouts production Synced Healthy argocd/aws-lbc-myapp-production-use1 myapp-production-use1 kube-system production Synced Healthy argocd/eso-myapp-production-use1 myapp-production-use1 external-secrets production OutOfSync Healthy ← known false positive argocd/eso-myapp-production-usw2 myapp-production-usw2 external-secrets production OutOfSync Healthy ← known false positive argocd/falco-myapp-dev-use1 myapp-dev-use1 falco production Synced Healthy argocd/falco-myapp-dev-usw2 myapp-dev-usw2 falco production Synced Healthy argocd/falco-myapp-production-use1 myapp-production-use1 falco production Synced Healthy argocd/falco-myapp-production-usw2 myapp-production-usw2 falco production Synced Healthy argocd/falco-myapp-staging-use1 myapp-staging-use1 falco production Synced Healthy argocd/falco-myapp-staging-usw2 myapp-staging-usw2 falco production Synced Healthy argocd/fluent-bit-myapp-dev-use1 myapp-dev-use1 logging production Synced Healthy argocd/fluent-bit-myapp-dev-usw2 myapp-dev-usw2 logging production Synced Healthy argocd/fluent-bit-myapp-production-use1 myapp-production-use1 logging production Synced Healthy argocd/fluent-bit-myapp-production-usw2 myapp-production-usw2 logging production Synced Healthy argocd/fluent-bit-myapp-

DevSecOps Pipeline — Completion Runbook All code is written and pushed to GitHub. This runbook covers the remaining operational steps: Terraform applies, GitOps ARN updates, and ArgoCD deployment. Prerequisites Install these tools if not already present: # AWS CLI v2 winget install Amazon.AWSCLI # Terraform 1.6+ winget install HashiCorp.Terraform # Terragrunt # Download from https://github.com/gruntwork-io/terragrunt/releases # Place in C:\Windows\System32\ or add to PATH # kubectl winget install Kubernetes.kubectl # ArgoCD CLI winget install argoproj.argocd AWS Profile Setup The root terragrunt.hcl uses profiles named myapp-{env}-{region_alias} . Configure them in ~/.aws/config : [profile myapp-production-use1] region = us-east-1 role_arn = arn:aws:iam::591120834781:role/AdministratorAccess source_profile = default [profile myapp-production-usw2] region = us-west-2 role_arn = arn:aws:iam::591120834781:role/AdministratorAccess source_profile = default [profile myapp-staging-use1] region = us-east-1 role_arn = arn:aws:iam::690687753178:role/AdministratorAccess source_profile = default [profile myapp-staging-usw2] region = us-west-2 role_arn = arn:aws:iam::690687753178:role/AdministratorAccess source_profile = default [profile myapp-dev-use1] region = us-east-1 role_arn = arn:aws:iam::557702566877:role/AdministratorAccess source_profile = default [profile myapp-dev-usw2] region = us-west-2 role_arn = arn:aws:iam::557702566877:role/AdministratorAccess source_profile = default PHASE 1 — Terraform Applies Work from the myapp-infra/ directory. Run in the order shown — capture outputs for updating GitOps files in Phase 2. 1.1 WAF (production + staging) # Production us-east-1 terragrunt apply --terragrunt-working-dir live/production/us-east-1/waf # Output → webacl_arn (copy this value) # Production us-west-2 terragrunt apply --terragrunt-working-dir live/production/us-west-2/waf # Output → webacl_arn (copy this value) # Staging (no GitOps ARN needed, but good to have) terra