标签：#cicd

Cloud teams waste between 40% and 60% of their infrastructure budget on a false choice: committing to reserved capacity they won't fully use or chasing spot instance savings they can't. Introduction: The Cloud Cost Optimization Dilemma Cloud teams waste between 40% and 60% of their infrastructure budget on a false choice: committing to reserved capacity they won't fully use or chasing spot instance savings they can't operationalize. The decision between commitment discounts and spot instances is not a preference. It is a calculation with three variables: workload predictability, failure tolerance, and the operational cost of managing interruptions. Commitment discounts lock you into capacity for one or three years. You pay upfront or monthly for compute resources whether you use them or not. The mechanism is simple: cloud providers offer 30% to 72% discounts because they can forecast their own capacity planning when customers commit. You save money when your actual usage matches your commitment. You lose money when usage drops below the committed level because you still pay for idle capacity. Spot instances offer 70% to 90% discounts by selling unused cloud capacity at auction prices. The provider can reclaim these instances with 30 seconds to 2 minutes of notice. You save money when your workload can tolerate interruptions and you build automation to handle instance termination. You lose money when interruptions cause failed jobs that must restart from scratch, consuming more compute time than the discount saved. Most engineering teams pick one strategy and apply it everywhere. This creates two failure modes. Teams that over-commit pay for capacity during low-traffic periods. Teams that over-rely on spot instances spend engineering time rebuilding checkpoint systems and retry logic that costs more than the discount delivers. The correct approach is workload-specific. Measure your actual usage patterns for 30 days. Calculate the cost of interruption handling. Then a

We wired our LLM eval suite into Datadog over about four months. Most of the panels we built got deleted. These are the five that stayed, and the metrics that feed them. TL;DR: We run an LLM-as-judge eval suite on every PR that touches a prompt, and we ship the results to Datadog as custom metrics. The dashboard started with fourteen panels. We kept five. The one that catches the most real regressions is per-criterion pass-rate split out by judge criterion, not the single rolled-up pass-rate number, because an aggregate of 91 percent hid the fact that one criterion had dropped from 0.95 to 0.62. Below are the metrics we emit, the Python that submits them, the monitor config we alert on, and the panels we tried and dropped. Some context on the setup so the rest makes sense. We are a Series-C dev-tool startup. We have a handful of prompts in production that do real work (classification, extraction, a summarization step in an agent loop). Each one has an eval set of tagged examples, somewhere between 80 and 400 per prompt. The judge is a separate model call that scores each output against a rubric. We run the suite in GitHub Actions. The eval job emits metrics to Datadog at the end of every run. Backend service health was already in Datadog, so putting eval data next to it meant one place to look during an incident instead of two. 1. Emit per-criterion pass-rate, not just the rolled-up number This is the one that earns its place. Our judge scores each output against multiple criteria. For the extraction prompt it is four: correct fields, no hallucinated fields, format valid, no refusal. Early on we only emitted one number, prompt_eval.pass_rate, the fraction of examples that passed every criterion. That number is fine for a smoke test and useless for debugging. The problem showed up on a prompt change that looked clean. Overall pass-rate went from 0.93 to 0.91. Two points. Nobody would block a PR on two points. But underneath, the "no hallucinated fields" criterion had

In Modern Time applications are no longer simple monolithic systems. Today organizations run: Microservices Kubernetes Containers Serverless Functions Multi-Cloud Platforms Distributed Systems As infrastructure becomes more distributed, troubleshooting becomes significantly harder. A single user request may travel through: Frontend ↓ API Gateway ↓ Microservice A ↓ Microservice B ↓ Database When something breaks, the biggest challenge becomes: "What exactly happened?" This is where Observability becomes critical. 🔗 Resources ** Support the Journey on GitHub: If you're following along, consider starring and forking the repo:** https://github.com/17J/30-Days-Cloud-DevSecOps-Journey What is Observability? Observability is the ability to understand the internal state of a system by analyzing the data it produces. In simple words: Can we understand what is happening inside our systems? Observability helps engineers answer: Why is the application slow? Which service is failing? Which request caused the issue? What changed recently? Where is latency occurring? Without observability: Problem Exists ↓ Guessing Begins With observability: Problem Exists ↓ Evidence Available ↓ Faster Resolution Why Observability Matters Modern cloud-native systems generate enormous amounts of data. Example: 100 Microservices ↓ Millions of Requests ↓ Thousands of Containers Traditional monitoring alone is no longer sufficient. Organizations need: Visibility Insights Correlation Root Cause Analysis Observability provides all of them. Monitoring vs Observability Many people confuse monitoring and observability. Monitoring asks: What is wrong? Observability asks: Why is it wrong? Example: Monitoring: CPU Usage = 95% Observability: Which service? Which request? Which dependency? Which deployment caused it? Observability provides context. The Three Pillars of Observability Modern observability is built on three primary pillars. Metrics Logs Traces Or: Monitoring Logging Tracing Together they provide a

7 Infra Improvement Strategies to Prevent Next.js Deployment Build Failures in 2026 Recently, our team's deployment pipeline started showing serious instability. Specifically, we encountered recurring build failures related to the chat build. As a result, the entire development team was preoccupied with battling these build failures. Attempts and Pitfalls Initially, I thought the --preload detection logic was the problem. I modified it to detect only specific lines, but this ended up causing issues in other areas. The recurring chat build failures were actually caused by the next.config file not properly recognizing file extensions. I modified it to allow extensions like .mjs , .js , .ts , and .cjs , but even that didn't work correctly at first, leading to some wasted effort. # .github/workflows/deploy.yml (Excerpt from initial version) - name : Run Preload Detection run : | # ... existing logic ... if [[ "$LINE" == *"some_pattern"* ]]; then echo "Preload detected" # ... fi I modified it to detect only specific lines like the above, which led to unintended behavior. // next.config.js (Initial configuration) module . exports = { // ... experimental : { // ... }, // ... }; Regarding extensions, I initially allowed only a few types, and only after experiencing chat build failures did I modify it to support more extensions. Root Causes In the end, it was a combination of several complex issues. There were flaws in the --preload detection logic, and the range of supported extensions in the next.config file was too narrow, which was the direct cause of the chat build failures. Additionally, there was confusion arising from the chat server builds being inconsistent between P1/P2 and P0 stages. Problems also occurred because the .next directory was not preserved, and the smoke gate was too lenient, failing to catch build failures. Finally, there was an unexpected side effect where the next/font/google library caused GCE outbound connection errors. Solutions To address these

Parent/child first. Evidence emission second. Ansible control plane third. Every release was a manual evidence collection exercise. The pipeline was the bottleneck. This is a redacted write-up of a real engagement: rebuilding a healthcare SaaS company's CI/CD pipeline across a fleet of Linux hosts on AWS. The context The engineering team had grown faster than the pipeline architecture had evolved. What started as a single-stage GitLab job for a small team had been extended, patched, and worked around as the team scaled past the patterns the original pipeline was built for. The result was familiar. Each deploy took 30 to 45 minutes of mostly-serial execution. Engineers had developed informal habits to work around the slowness, including pushing partial changes outside the pipeline when the timeline got tight. Audit windows were preceded by three-week sprints in which the team manually compiled deployment logs, screenshots of access reviews, and approval chains into PDFs describing what the pipeline was supposed to be doing. The work was technically passing HIPAA audits, but the audit was a snapshot of a system the auditor could not independently verify. The cost was paid twice: the velocity loss on every deploy, and the three-week scramble before each assessment. The team knew the architecture was wrong. They needed engineering hands to redesign it without slowing the product roadmap the audits were already eating into. The approach The redesign moved in three layers. First, decompose the monolithic pipeline into parent/child stages so work can parallelize and the audit boundary of each stage is provable. Second, build structured evidence emission into every stage as a property of how it runs, not an after-the-fact compilation task. Third, layer an Ansible control plane across the host fleet so HIPAA control state is continuously validated, not reviewed quarterly. ┌────────────────────────── Parent Pipeline (.gitlab-ci.yml) ──────────────────────────┐ │ │ │ ┌────────

The first time I deployed a Next.js app to production, it took me three days. Not because the app was complicated — it was a straightforward portfolio site. It took three days because I had no idea what I was doing with AWS, I'd never written a GitHub Actions workflow, and every tutorial I found either skipped the hard parts or assumed I already knew them. By the time I was done, I had a deployment pipeline I was genuinely proud of: push to main, GitHub Actions runs the build, tests pass, the app deploys to an EC2 instance behind CloudFront. Zero manual steps. Zero downtime deploys. Total cost: about $5/month. This guide is the one I wish had existed. We're going to deploy a Next.js app to AWS from scratch — EC2 for compute, CloudFront for CDN, GitHub Actions for CI/CD — with every step explained so you understand what you're building, not just copying commands. Why AWS Instead of Vercel? This is a fair question. Vercel is genuinely excellent for Next.js, and for most projects it's the right call. You push, it deploys. Done. AWS makes sense when: You need to control the infrastructure (compliance, data residency, custom VPC configuration) You're running other services (databases, queues, lambdas) in AWS and want everything in the same network You want to learn infrastructure skills that transfer to enterprise environments Your app has specific performance requirements that benefit from custom CloudFront configuration You're a freelancer or consultant who wants to bill separately for infrastructure If none of those apply to you, use Vercel. This guide is for when they do. The Architecture Here's what we're building: ┌────────────────────────────────────────────────────────┐ │ GITHUB ACTIONS CI/CD │ │ │ │ Push to main → Build → Test → Deploy to EC2 │ └──────────────────────┬─────────────────────────────────┘ │ SSH deploy ▼ ┌────────────────────────────────────────────────────────┐ │ AWS EC2 INSTANCE │ │ │ │ Ubuntu 22.04 LTS │ │ Node.js 20 + PM2 (process manager) │ │ N

Appendix: Live System Output — Real Pipeline in Production All output below was captured live from the running pipeline on 2026-03-08. These are not mock outputs — they come from actual AWS infrastructure and Kubernetes clusters. ArgoCD — All 50 Applications Across 6 Clusters The following is the live output of argocd app list from the hub cluster ( myapp-production-use1 ). Every component of the pipeline is represented — security, logging, monitoring, backups, and the application itself. $ argocd app list --output wide NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH argocd/argo-rollouts-myapp-production-use1 myapp-production-use1 argo-rollouts production Synced Healthy argocd/argo-rollouts-myapp-production-usw2 myapp-production-usw2 argo-rollouts production Synced Healthy argocd/aws-lbc-myapp-production-use1 myapp-production-use1 kube-system production Synced Healthy argocd/eso-myapp-production-use1 myapp-production-use1 external-secrets production OutOfSync Healthy ← known false positive argocd/eso-myapp-production-usw2 myapp-production-usw2 external-secrets production OutOfSync Healthy ← known false positive argocd/falco-myapp-dev-use1 myapp-dev-use1 falco production Synced Healthy argocd/falco-myapp-dev-usw2 myapp-dev-usw2 falco production Synced Healthy argocd/falco-myapp-production-use1 myapp-production-use1 falco production Synced Healthy argocd/falco-myapp-production-usw2 myapp-production-usw2 falco production Synced Healthy argocd/falco-myapp-staging-use1 myapp-staging-use1 falco production Synced Healthy argocd/falco-myapp-staging-usw2 myapp-staging-usw2 falco production Synced Healthy argocd/fluent-bit-myapp-dev-use1 myapp-dev-use1 logging production Synced Healthy argocd/fluent-bit-myapp-dev-usw2 myapp-dev-usw2 logging production Synced Healthy argocd/fluent-bit-myapp-production-use1 myapp-production-use1 logging production Synced Healthy argocd/fluent-bit-myapp-production-usw2 myapp-production-usw2 logging production Synced Healthy argocd/fluent-bit-myapp-

DevSecOps Pipeline — Completion Runbook All code is written and pushed to GitHub. This runbook covers the remaining operational steps: Terraform applies, GitOps ARN updates, and ArgoCD deployment. Prerequisites Install these tools if not already present: # AWS CLI v2 winget install Amazon.AWSCLI # Terraform 1.6+ winget install HashiCorp.Terraform # Terragrunt # Download from https://github.com/gruntwork-io/terragrunt/releases # Place in C:\Windows\System32\ or add to PATH # kubectl winget install Kubernetes.kubectl # ArgoCD CLI winget install argoproj.argocd AWS Profile Setup The root terragrunt.hcl uses profiles named myapp-{env}-{region_alias} . Configure them in ~/.aws/config : [profile myapp-production-use1] region = us-east-1 role_arn = arn:aws:iam::591120834781:role/AdministratorAccess source_profile = default [profile myapp-production-usw2] region = us-west-2 role_arn = arn:aws:iam::591120834781:role/AdministratorAccess source_profile = default [profile myapp-staging-use1] region = us-east-1 role_arn = arn:aws:iam::690687753178:role/AdministratorAccess source_profile = default [profile myapp-staging-usw2] region = us-west-2 role_arn = arn:aws:iam::690687753178:role/AdministratorAccess source_profile = default [profile myapp-dev-use1] region = us-east-1 role_arn = arn:aws:iam::557702566877:role/AdministratorAccess source_profile = default [profile myapp-dev-usw2] region = us-west-2 role_arn = arn:aws:iam::557702566877:role/AdministratorAccess source_profile = default PHASE 1 — Terraform Applies Work from the myapp-infra/ directory. Run in the order shown — capture outputs for updating GitOps files in Phase 2. 1.1 WAF (production + staging) # Production us-east-1 terragrunt apply --terragrunt-working-dir live/production/us-east-1/waf # Output → webacl_arn (copy this value) # Production us-west-2 terragrunt apply --terragrunt-working-dir live/production/us-west-2/waf # Output → webacl_arn (copy this value) # Staging (no GitOps ARN needed, but good to have) terra