标签：#sre

As a site reliability engineer in a global company, I'm running a modern (well, relatively modern, to be honest and modest) cloud-native stack: HashiCorp Vault as the secret manager, workloads on Kubernetes clusters in AWS (EKS), and development workflows automated through Jenkins (legacy) and GitLab CI. This setup is, quite likely, familiar to you — it's the normal playbook in the cloud-native era. In theory, we have the right tools for both security and efficiency: After all, we have a state-of-the-art secret manager integrated with everything. But in reality, it's far from the truth. See if you resonate with the following scenarios: Scenario A: A new colleague just joined the team. Manager: "Your initial password to log in to your corporate account came to me via email, but since you can't log in to your mail account just yet, here, take a picture of my screen." (In some companies, taking a picture of a computer monitor would get you fired, I'm not kidding.) Scenario B: A developer needs a temp password to access a database. Dev: "Where is the newly created temporary password? Need it for debugging." Ops: "In the Vault." Dev: "I can't access Vault." Ops: "No, you can't. It's not safe to open UI access to Vault. Corporate policy." Dev: "Then how can I get the password?" Ops: "Well... Technically, the password isn't in the Vault. There is a Jenkins pipeline that calls the Vault API to generate a temp password, then stores it in Jenkins secrets. You need to request access to the corresponding Jenkins pipeline, trigger it, then get the secrets from Jenkins." Dev: "Why on earth do we store secrets in Jenkins when we have Vault, which we aren't allowed to use?" Ops: "Corporate policy, just told you." Scenario C: A new ops team member needs to update a certificate for a service running in production for the first time. Ops: "Where is the old cert?" Mentor: "In K8s as a secret." Ops: "Where is the cluster?" Mentor: "In AWS." Ops: "How do I access that?" Mentor: "You need

I've built dozens of dashboards. Most have been ignored. A few have been used constantly. The difference isn't the graphs. It's the design. The 3-second test A useful dashboard answers 'is everything OK?' in 3 seconds. Not 'let me scroll through 40 graphs to find out.' Big colored header at the top: green = healthy, yellow = watching, red = broken. That's the 3-second answer. Everything else is drill-down. The hierarchy rule Three layers, no more: Overview — one line per service, status color, key SLI Service detail — one dashboard per service, 6-12 graphs max Deep dive — triggered from service detail, domain-specific Anything beyond 3 layers is 'please get lost in my dashboard tree.' The on-call test Imagine you're on-call at 3 AM. You get paged for 'service X is slow.' Can you, in 30 seconds, use this dashboard to tell if the problem is the service itself, its database, its upstream dependency, or its downstream consumers? If yes, the dashboard works. If no, redesign. What to cut Graphs with no baseline (flat line or spiky forever — how do you know if it's bad?) Metrics you've never used in an actual incident Vanity metrics (total requests ever) Graphs where the y-axis is in units nobody understands The hidden metric The real measure of a dashboard's value: does the on-call engineer open it before or after the paging tool? If they open it first — it's their compass. If they open it only after being paged — it's a reference, not a dashboard. Aim for the first. Written by Dr. Samson Tanimawo BSc · MSc · MBA · PhD Founder & CEO, Nova AI Ops. https://novaaiops.com

Liquid syntax error: Variable '{{service=\"{service}' was not properly terminated with regexp: /\}\}/

I’ve seen enough postmortems to know that a large percentage of them are essentially polite fiction. We sit in a meeting, everyone is exhausted from the outage, and we agree on a narrative. We write down that 'the database connection pool was exhausted' or 'a bad deploy caused an error spike.' Then we add an action item like 'add more monitoring' or 'improve testing,' and we move on to the next feature request. Three months later, the exact same thing happens. The same service, the same error, the same fatigue. We didn’t solve anything; we just documented our failure with slightly better prose. The problem is that most postmortems fail at the fundamental level of investigation. They stop at the symptoms. They treat human error as a root cause—which is lazy engineer shorthand for 'we don't want to fix the system.' And they treat vague timelines as acceptable data, which makes reconstruction impossible when you’re trying to correlate logs from different subsystems. This is why I became interested in using MCP (Model Context Protocol) not just to give agents access to my tools, but to act as an auditor for these processes. Most people use LLMs to summarize what happened. That's useless. You don't need a summary; you need someone to tell you where your investigation is weak. I’ve been working with the Incident Postmortem Prover ( https://vinkius.com/mcp/incident-postmortem-prover ) because it doesn't try to be a scribe. It acts as an adversarial auditor for SREs and engineers. It uses semantic trap lists designed to catch exactly the kind of hand-wavy logic that ruins investigations. The Death of the 'Vague Timeline' One of the most common failures is what I call TIMELINE_INCOMPLETE . You'll see things like: 'Around 3 PM, we noticed a spike in errors. By 4 PM, everything was back to normal.' That isn't a timeline; it's an anecdote. An actual investigation needs minute-by-minute reconstruction in UTC. What happened at 15:02? Who acknowledged the PagerDuty alert? When did

TL;DR: Passing a historical blockNumber to eth_call , eth_getBalance or eth_getLogs silently routes your request to the archive tier of hosted RPC providers. In our production metrics, archive calls cost on average 26.7x more compute units than the same call at latest . This post explains why, shows the exact client code pattern that triggers it, and gives you three Prometheus queries to measure your own archive exposure in under a minute. Full cross provider measurements are published in the OpenChainBench RPC benchmarks . Last week our RPC cost dashboard flagged an overage projection above four thousand dollars for a single billing cycle on a single provider. On paper, our services were doing normal eth_call operations. In practice, one small pattern buried in three separate indexers had multiplied our compute unit consumption by more than an order of magnitude, and nothing in the code review process had surfaced it. This post breaks down what an archive multiplier is, why it silently inflates blockchain RPC bills across every major hosted provider, and how to detect it in your own Prometheus stack before the next overage alert lands in Slack. What does "archive" mean at the Ethereum node level? Every request that reads the state of a smart contract, whether through eth_call , eth_getBalance , eth_getStorageAt , eth_getCode , or a batch of these, requires the RPC node to reconstruct the world state at a specific block height. Ethereum clients handle this in two modes. Full node mode. The state trie is kept in memory or on fast SSD for the tip of the chain plus a rolling window of recent blocks. On Geth default settings that window is 128 blocks deep. Any query targeting latest , pending , or a block within that window resolves in a few milliseconds against the current state. Archive node mode. The client preserves every intermediate state trie since genesis. Answering a query at a block from months or years ago requires reading historical trie data off disk and re

Over the last few years, I've been exploring AI agents, and one thing became obvious. There are hundreds of AI agents available today, but almost all of them are general-purpose. They can answer questions, write code, or browse the web, but very few truly understand the day-to-day challenges of running production infrastructure. As someone who has spent years working in DevOps, I wanted something different. That's why I built DevOps Open Agent, an open-source, self-hosted AI platform designed specifically for DevOps engineers, SREs, and Platform teams. Today, the project includes: ✅ Kubernetes Debugging Agent for AI-assisted cluster troubleshooting ✅ AWS DevOps Agent for investigating infrastructure issues ✅ Cloud Cost Detector to identify optimization opportunities ✅ GitHub PR Reviewer with DevOps-focused code reviews ✅ Slack, Microsoft Teams, and PagerDuty integrations ✅ MCP support for connecting external tools and services ✅ Support for multiple LLM providers including OpenAI, Anthropic, Gemini, OpenRouter, and Ollama But this is just the beginning. There is so much more we can build together: ✔️ Better Kubernetes diagnostics ✔️ Smarter AWS investigations ✔️ Terraform and Infrastructure-as-Code analysis ✔️ Observability integrations ✔️ Performance debugging ✔️ Security analysis ✔️ Historical investigation memory And many more AI-powered workflows for production engineering If you're passionate about DevOps, SRE, Platform Engineering, or Generative AI, I'd love to have you involved. Whether you contribute code, improve documentation, report bugs, review pull requests, or suggest new ideas, every contribution helps move the project forward. ⭐ Give the repository a star 🍴 Fork the project 🚀 Pick an issue and submit a pull request If you've been looking for an opportunity to work at the intersection of DevOps and AI, this is it. Let's build the open-source AI platform that every DevOps engineer wishes existed. 🔗 Repository: https://github.com/ideaweaver-ai/devops-op

Build software that heals itself in the agentic era Bucabay Bucabay Bucabay Follow Jul 1 Build software that heals itself in the agentic era # ai # agents # architecture # security 13 reactions 3 comments 10 min read

Circuit breakers and bulkheads are design patterns — their numbers are operational weapons. Failure ratio 50% or 30%? Max concurrent calls 25 or 100? During an outage the right answer changes hourly . Code the pattern once; tune thresholds live . Kiponos.io separates resilience structure (in Java) from resilience parameters (in live config tree). Pattern in code, numbers in Kiponos public boolean allowCall ( String downstream ) { var cfg = kiponos . path ( "resilience" , downstream ); return breaker ( downstream ) . failureRateThreshold ( cfg . getFloat ( "failure_rate_threshold" )) . waitDurationInOpenState ( cfg . getInt ( "open_seconds" )) . permittedInHalfOpen ( cfg . getInt ( "half_open_calls" )) . tryAcquire (); } Ops opens circuit sensitivity during brownout — dashboard edit, not redeploy. Resilience tree resilience/ payments-api/ failure_rate_threshold : 0.5 open_seconds : 30 half_open_calls : 5 bulkhead_max_concurrent : 40 inventory-api/ failure_rate_threshold : 0.35 open_seconds : 60 bulkhead_max_concurrent : 25 global/ force_open_all : false Extreme: coordinated degradation Platform SRE sets force_open_all: false normally. During regional disaster, flip selective open_seconds sky-high on non-critical downstreams — bulkhead by configuration , Java still executes pattern logic. Performance Breaker checks are per-call — getFloat() must be local. See rate limits article . Getting started Externalize Resilience4j YAML values to resilience/* Incident drill: tighten failure_rate_threshold live Resources: github.com/kiponos-io/kiponos-io Kiponos.io — resilience patterns with live numbers. Breakers that bend during the outage.

The Post-Mortem Nobody Learns From I've sat through hundreds of post-mortems. Most follow the same pattern: something breaks, someone writes a Google Doc, we have a meeting, we list action items, nobody follows up, the same thing happens again in 3 months. Here's how to break the cycle. The Blameless Culture Trap "Blameless" doesn't mean "actionless." The biggest failure mode I see is teams that use blameless culture as an excuse to avoid accountability. Blameless means: we don't punish the person who pushed the bad deploy. Blameless does NOT mean: nobody is responsible for fixing the systemic issue. My Post-Mortem Template # Incident: [SERVICE] [SYMPTOM] on [DATE] ## Impact - Duration: X minutes - Users affected: N - Revenue impact: $X - SLO budget consumed: X% ## Timeline (UTC) - HH:MM - First alert fired - HH:MM - On-call acknowledged - HH:MM - Root cause identified - HH:MM - Fix deployed - HH:MM - Service recovered - HH:MM - All-clear declared ## Root Cause [2-3 sentences. Technical but readable.] ## Contributing Factors 1. [Factor that made the incident possible] 2. [Factor that made detection slow] 3. [Factor that made resolution slow] ## What Went Well - [Something that worked] - [Something that helped] ## What Went Wrong - [Process failure] - [Technical gap] ## Action Items | Action | Owner | Priority | Due Date | Status | |--------|-------|----------|----------|--------| | ... | ... | P1/P2/P3 | ... | Open | ## Lessons Learned [1-2 paragraphs of genuine insight] The Action Item Problem Action items from post-mortems have a 30% completion rate industry-wide. That's terrible. Here's why: Too many items (I've seen post-mortems with 15 action items) No clear ownership No deadline No follow-up mechanism Competing with feature work The Fix: Three Rules Rule 1: Maximum 3 action items per post-mortem. If you can't narrow it to 3, you haven't identified the real problems. Rule 2: Every action item gets a JIRA ticket linked to the next sprint. Not "someday." Not "bac

Table of Contents The Moment That Changed Everything What It Actually Is The Setup Nobody Believes Is This Simple Focused Sessions — One Agent, One Mission Sub-Agents With Specific Skill Sets Building MCPs for Your Own Stack What We've Actually Achieved How It Became a Force Multiplier for Incident Response From OpenCode to FRIDAY — The Agent That Investigates Incidents Autonomously From FRIDAY to JARVIS — Thinking About Write-Path Autonomy Deleting 130,000 Accounts Without Writing Code Learning the Entire Product in Conversations Why This Is Different From ChatGPT The Uncomfortable Truth What's Next The Moment That Changed Everything It was 11pm on a Tuesday. A cache migration in our production environment had just caused thousands of authentication failures for two of our largest enterprise customers. Our VP of Product wanted answers. Our support team was fielding escalations. And our engineers were alt-tabbing between AWS console, Datadog, GitHub, Azure DevOps, and PagerDuty trying to piece together what happened. Three weeks later, when we needed to attempt the same change again, an engineer typed this into a terminal: "Review the ADO change ticket, compare the MOP against the actual ElastiCache configuration in prod region, check the K8s config repo for how Redis env vars are wired on the Green cluster, and tell me if this approach avoids the token validation failure that caused the previous customer impact." Fourteen seconds later , the system had pulled the work item, queried AWS ElastiCache across four regions, read the Kubernetes configuration from GitHub, cross-referenced the deployment patches, and delivered a precise technical assessment including a risk it identified that the team hadn't documented: in-flight tokens during the 30–60 second Global Accelerator propagation window. That system is OpenCode — an AI-powered CLI assistant connected to our entire operational stack through the MC(Model Context Protocol). And it has fundamentally changed how a 20-

I run a paid infrastructure service. Alone. No co-founder, no on-call rotation, no senior engineer to escalate to. My only collaborator is Claude Code, and after about a year, my persistent memory has grown to 60+ entries. Those entries have become more valuable than any runbook I've written. They've also taught me — painfully — what makes memory architecture work and what makes it quietly fail. If you're running anything solo with an AI agent, here are five lessons I wish I'd burned into my brain on day one. 1. Write the why , not the what The first instinct when you start using persistent memory is to log what you did. "Migrated service X from tool A to tool B." "Switched protocol from X to Y." Six months later, when something breaks, that information is worthless . You don't need to know what you did — git log and git blame already tell you that. You need to know why you made that choice. What constraint forced it. What you ruled out. Real example. The bad version of an entry I once wrote: Switched the worker pool from Docker containers to systemd units on host. Tells me nothing my git history doesn't. The rewritten version: systemd units on the host instead of Docker containers on this VPS provider. Why: the provider runs aggressive kernel-wide OOM scoring across tenants; containers were getting reaped by oom-killer triggered by other customers' workloads. systemd processes survive because they're scored as system processes. How to apply: any VPS where dmesg | grep -i oom shows kills from PIDs you don't recognize — don't run containers there, run systemd. That one entry has saved me three rebuilds. Because the next time I'm tempted to "just dockerize it, it'll be cleaner," the memory entry says: no, you already learned this, you'll be back here in a week. The pattern: always include Why: and How to apply: lines. If a memory entry can't answer those two questions, delete it. 2. Memory rots — prune or pay About six months in, I did a memory audit. Of 60 entries, 1

Chaos engineering sounds expensive. Netflix built Chaos Monkey to randomly kill production servers. Google runs DiRT (Disaster Recovery Testing) across their entire infrastructure. Amazon does game days where they intentionally take down services. You're building a Node.js API. You don't have a platform team. You don't have a chaos infrastructure. But you still need to know: what happens when your dependencies get slow? The good news is that 80% of the value of chaos engineering comes from one question, and you can answer it locally in five minutes. The one question that matters What does my application do when a dependency responds slowly or not at all? Not "what if the server catches fire" — that's infrastructure chaos. What about application chaos: the database is slow, the payment API is timing out, Redis is having a bad day. These happen constantly in production and they're almost never tested. The failure modes look like this: Your DB gets slow under load → your API response times climb → your timeout fires → you retry → now you're sending twice the load to an already-slow DB Your Redis cache goes down → every request hits Postgres directly → Postgres gets slow → same cascade Stripe's API takes 3 seconds instead of 200ms → your checkout route times out → users get errors → you're losing revenue Every one of these is a latency failure , not a crash. The service is still up. It's just slow. And slow is the hardest failure mode to test because your local environment is fast. Why local testing misses this When you test locally, your "database" is either: A real local Postgres running on the same machine (sub-millisecond latency, not production-like) A mock that returns instantly ( jest.fn().mockResolvedValue(data) ) A fake with a flat delay ( await sleep(200) ) None of these produce realistic latency. A real production database has: Fast responses most of the time (p50 ~5ms) Occasional slowdowns (p95 ~50ms) Rare but real spikes (p99 ~200ms, p99.9 ~2000ms) The spik

A model fallback that only works in a diagram is not resilience. It is a TODO with better branding. If your product depends on AI agents, one slow provider, rate-limit spike, regional restriction, malformed response, or model behavior change can turn a useful workflow into a confusing user experience. The dangerous part is not always a clean outage. The dangerous part is a half-working fallback that silently changes schemas, drops tool state, skips citations, or gives users lower-confidence output without saying so. This guide shows how to run practical AI model failover drills before production traffic teaches you the lesson the hard way. The goal is not to make every model interchangeable. The goal is to keep the user workflow safe, honest, and recoverable when the primary model cannot do the job. Why model failover needs drills, not just retries Most teams start with a simple fallback chain: try the primary model, then a backup model, then show an error. That is better than nothing, but it misses the real problems in AI applications. Traditional APIs usually fail in obvious ways: timeout, 500, bad credentials, quota exceeded. AI systems can fail more subtly: The backup model returns valid JSON with different field meanings. A cheaper model ignores part of the tool policy. A provider accepts the request but streams tokens too slowly. A fallback model does not support the same function-calling format. A regional policy or access rule changes availability. The model completes the answer but loses citation discipline. The agent retries and burns the tenant budget. The final response looks polished but skipped the expensive verification step. Recent AI infrastructure conversations are pointing in the same direction: the system around the model now matters as much as the model. Agent benchmarks, provider reliability, AI cost pressure, and model routing are all active developer concerns. Search results also show many broad posts about LLM fallback strategy, but fewer pr

The Runbook That Lied to Me at 3am The pager went off at 3:14am for a wedged OpenStack Neutron agent. I did what any tired engineer does: I opened the runbook. It told me to restart a service that had been renamed eighteen months earlier, pointed at a Grafana dashboard that 404'd, and assumed a network topology we'd migrated off of two quarters back. The runbook wasn't just unhelpful. It was actively lying to me, and I burned twenty minutes trusting it before I gave up and went to read the source. That's the real problem with documentation. It isn't that we don't write it. It's that the moment we finish writing it, it starts rotting, and the cost of keeping it fresh is high enough that nobody pays it until the document has already betrayed someone at 3am. A runbook your team doesn't trust is worse than no runbook, because no runbook at least forces you to think. This is where AI actually earns its keep in a platform org, and not in the way the marketing decks suggest. AI is not going to own your documentation. It's going to do the tedious first-draft labor — turning a resolved incident, a chunk of shell history, or a deploy diff into a structured skeleton — so a human engineer can spend their scarce attention on the part that matters: verifying the commands, marking what's unproven, and editing the robotic tone out so the team actually reads it. AI drafts. You verify and sign off. That distinction is the whole game. Why "Humanizing" AI Is the Job, Not a Slogan Let me be precise about what I mean by "humanizing AI," because the phrase gets abused. I don't mean making AI sound human to fool a reader. I mean keeping a human in the loop as the editor and owner of record, and doing the unglamorous work of turning a competent-but-soulless machine draft into something a colleague trusts. Two things break trust in AI-drafted docs, and both are fixable by a human pass: Unverified claims stated with confidence. An LLM will happily tell you to run systemctl restart neutron-l3-

Your runbooks are out of date. I don't know your team, but I'd bet money on it. Most teams write runbooks once, in a panic after an outage, and then never touch them again until the next outage proves them wrong. How runbooks rot Steps reference deprecated tools. The grafana dashboard moved, the CLI command was renamed, the bastion host got retired. Nobody updated the runbook because nobody re-ran it during the calm months. The team that owned the system left. Three of the five engineers who wrote it are gone. The remaining two haven't actually run the runbook in 18 months because the outage type it covers stopped happening. Half-truths from the start. The original author skipped the obvious steps (because they were obvious to them) and the new on-call engineer can't reproduce the recovery. The result: at 2 AM during the actual incident, the runbook sends you down a dead end. Now you're improvising under stress. What working runbooks have in common I've seen exactly two patterns work: 1. Runbooks live with the code. Put them in the repo of the system they document. When the code changes, the runbook PR is part of the same review. Out-of-repo wikis die first because the cognitive distance is too great. 2. Runbooks are reviewed by people who weren't there. Have a junior engineer run through the runbook quarterly on a non-incident day. Every place they get stuck is a real bug in the document. Fix it then. The author will be too close to see the gaps. The three sections that matter A useful runbook has exactly three sections: Symptom : how do I know this is the right runbook? Concrete signals, with example screenshots if visual. First 5 minutes : what to do RIGHT NOW to stop the bleed. Not the root cause investigation, just the triage actions. Investigation : where to look, what queries to run, what to escalate. That's it. Anything else (architecture diagrams, history, philosophy) goes in a separate doc that the runbook links to. The cultural part The hardest part isn't

Failures are obvious. Retries are sneaky. When something fails, everyone notices. An alert goes off. A request errors out. Someone starts investigating. Retries are different. They look harmless. Most of the time, they save the system. But sometimes, retries create bigger problems than the original failure. Imagine an API call times out. No problem. The system retries. But what if the first request actually succeeded and only the response was lost? Now the retry creates: duplicate orders repeated emails inconsistent records workflows running twice The failure happened once. The retry multiplied it. Another thing I've seen: One slow dependency causes requests to pile up. Retries start firing. Those retries create even more traffic. Which slows things down further. Which triggers even more retries. Suddenly, the system is spending more effort retrying than doing useful work. Retries also hide problems. A temporary issue gets retried five times and eventually succeeds. Everything looks normal. Meanwhile: latency increases queues grow users experience delays Nothing technically failed. But the system is getting less healthy. What changed for me is that I stopped treating retries as free. Every retry has a cost. It consumes resources. It increases load. And if actions aren't designed carefully, retries can repeat side effects that should only happen once. Now when I build something, I don't ask: "What happens if this fails?" I ask: "What happens if this runs again?" Because in production, things almost always run again. And if the answer is "bad things happen," the retry mechanism isn't helping. It's making things worse. Failures are part of every system. Retries are too. The difference is that failures usually happen once. Retries can turn one problem into hundreds if you don't design for them. This is something we think about constantly at BrainPack when operating long-running workflows across multiple systems. AI and automation layers make retries even more common, wh

Most AI coding tools assume you're sitting in front of a repo. There's a working directory, some source files, tests, maybe a CI pipeline. The AI reads your code, suggests changes, ru****ns tests. Great model — if you're a developer writing code on a Tuesday afternoon. Now picture the other scenario. It's 2am. PagerDuty fires. You SSH into a box you haven't touched in three months. Something is broken, you're not sure what, and the runbook was last updated by someone who left the company in 2022. You're not thinking about repos. You're thinking: what OS is this thing running? What just failed? Is it safe to restart that service or will I make it worse? These are two fundamentally different workflows. But almost every AI terminal tool I've seen is built for the first one. The 30 minutes nobody builds for There's a ton of tooling for the world before you log into the box: Prometheus, Grafana, PagerDuty, incident.io, runbooks, dashboards. All useful. No complaints. But there's this practical 30-minute window after you SSH in where you're basically doing archaeology with journalctl and grep. You check systemd. You look at disk. You read logs that were clearly written by someone who hated future-you. You copy-paste terminal output into Slack so your teammate can squint at it from a different timezone. This is where I think AI could actually help. Not by replacing Grafana. Not by building another dashboard. Just by being present in the shell while you're debugging. Please, for the love of uptime, don't replace my shell I've seen a few AI terminal projects that basically build an entire new terminal experience from scratch. New keybindings, new UI, new everything. Here's the thing: ops people have muscle memory. We have aliases we wrote in 2019 and forgot about. We have tmux configs we'd defend with our lives. We SSH through jump hosts with key forwarding chains that barely work but have worked for years so nobody touches them. If your AI tool requires me to abandon all of

Why This Matters: The 2 AM Problem It's 2 AM. Your phone rings. Your production database is down. Customers can't log in. Revenue is dropping by the second. You call the Ops team. They restart the server. Downtime: 45 minutes. Cost: $100K in lost sales. Root cause? Unknown. This happens thousands of times a week at companies worldwide. The question isn't "Will your system break?" It's " When it breaks, are you ready? " That's where SRE comes in. What is SRE? (The Real Definition) SRE (Site Reliability Engineering) = Applying software engineering principles to build reliable, scalable infrastructure and systems. It's not just about keeping servers running. It's about: Reliability : Systems that don't break unexpectedly Scalability : Systems that handle growth without collapsing Infrastructure : Automating how systems are built, deployed, and monitored Measurability : Knowing exactly how your system is performing at any moment Traditional operations manages infrastructure reactively — when something breaks, you fix it. SRE manages infrastructure proactively — you engineer it so it rarely breaks, and when it does, it heals itself. The Key Insight: Reliability is Engineered, Not Hoped For Here's the critical shift in thinking: Old mindset : "Let's build this system and hope it doesn't break." SRE mindset : "Let's measure what 'reliable' means, design the system to achieve that, and automate the monitoring and recovery." But reliability isn't just uptime. It includes: Uptime : Is the system available? Latency : How fast does it respond? (A slow system is effectively broken) Error rate : What percentage of requests fail? Throughput : Can it handle the traffic? User experience : Does the system meet user expectations? All of these are engineered and measured. A Simple Analogy: The Bridge Imagine you're managing a bridge. Traditional approach : Engineers patrol daily, react to problems, work around the clock fixing issues SRE approach : Engineers design monitoring that aler

The thesis Quorum is built on is uncomfortable and true: the tools a team uses to coordinate an incident often live in the same region as the thing that is failing. When the region goes, the incident response goes with it. You are now coordinating a region outage over a status page that the region outage took down. Quorum is an incident command plane designed to survive a region loss. This post is about how the failover works, what the live demo does and does not prove, and where the survival story currently ends, because a database audience will ask all three and they deserve a straight answer. What DSQL gives you A multi-region DSQL cluster in the US set is three regions: two full regions, which for Quorum are us-east-1 and us-east-2, and a log-only witness in us-west-2 that has no cluster endpoint of its own. Both full-region endpoints present a single logical database with strong consistency, and the architecture is designed for 99.999% multi-region availability with no single point of failure and automated failure recovery . The behavior that matters for an incident tool is stated plainly in the GA announcement : applications can keep reading and writing with strong consistency even when they are unable to connect to a region's cluster endpoint, and the third region acts as a log-only witness with no cluster resource or endpoint. The survivor keeps serving; the witness holds the log so the surviving region keeps commit quorum. Quorum is, in effect, a live demonstration of that reference behavior with an incident-command product wrapped around it. Quorum's failover layer AWS's guidance for multi-region DSQL is to put routing in front of the endpoints: either DNS-based routing with Route 53, or application-level routing logic, so traffic redirects automatically when an endpoint becomes unreachable. This is laid out in Implement multi-Region endpoint routing for Amazon Aurora DSQL . Quorum, a Next.js app on Vercel, does the application-level version: it detects an

If you're a DevOps, SRE, or Cloud engineer in India — or hiring one — the market in 2026 looks very different from a few years ago. Instead of guessing, we analyzed 800+ live DevOps/SRE/Cloud/Platform Engineering roles currently on PuneOps to see what's actually being hired for right now. Here's what the data shows. 1. Bangalore dominates, but the market is genuinely national DevOps hiring in India is no longer a one-city story. Of the live roles: Bangalore — the clear leader, ~25% of all listings Pune — a strong #2 (and a serious DevOps hub, not just an IT-services town) Hyderabad, Mumbai, Delhi NCR, Chennai — all with steady, healthy demand Remote / Pan-India — roughly a third of all roles don't tie you to a city at all Takeaway for candidates: you're no longer limited to wherever you live. Remote and pan-India DevOps roles are a huge and growing slice of the market. 2. This is a senior-heavy market The single most striking pattern: DevOps hiring in India skews experienced. The largest band by far is 5–10 years of experience A meaningful chunk wants 10+ years (architects, principals, platform leads) Entry-level (0–2 years) roles are comparatively rare Takeaway: DevOps remains a hard field to break into directly. Most roles assume you've already done software, sysadmin, or cloud work. If you're junior, the path in is usually via a software/ops role, then specializing. 3. The skills employers actually ask for Across the listings, the same technologies show up again and again: Kubernetes — effectively table stakes now Terraform / IaC — infrastructure-as-code is expected, not bonus AWS / Azure / GCP — cloud fluency, often multi-cloud CI/CD pipelines, observability, and Python for automation Takeaway: if you're leveling up, Kubernetes + Terraform + one major cloud is the core combination Indian employers are screening for in 2026. 4. Salary ranges (market benchmarks, 2026) Compensation varies widely by company type (product vs. services), city, and exactly how senior t