标签：#Reliability

Zhou Yu discusses why AI agents stall in demo phase and shares how simulation-driven testing solves compliance and reliability bottlenecks. Learn how Columbia and Arklex AI use synthetic user personas, trajectory entropy, and automated CI/CD pipelines to evaluate multi-turn agents, catch edge cases before deployment, and scale self-learning workflows in production. By Zhou Yu

Microsoft has enabled zone redundancy on the Standard v2 tier of Azure API Management, following its arrival on Premium v2 in December. Standard v2 starts at $700 per month against $2,801 for Premium v2, but carries a 99.95% SLA rather than 99.99%. Zone redundancy can only be configured when creating an instance. By Steef-Jan Wiggers

Warum KI-Agenten Leitplanken brauchen: Operatives Gedächtnis statt Over-Engineering Ki-Agenten sind nicht böse. Sie sind nicht einmal unzuverlässig im klassischen Sinne. Das eigentliche Problem ist vielmehr ihre beständige Bereitschaft zu helfen, gepaart mit einem fehlenden Verständnis für die Grenzen ihrer Befugnisse. Sie wollen das Problem lösen, das ihnen gestellt wird, oft mit einer Aggressivität, die menschliche Manager selten aufbringen. Wenn ein Agent eine Produktionsdatenbank bereinigen soll, tut er es. Wenn er eine Datei löschen soll, die er für überflüssig hält, weil sie im aktuellen Kontext nicht erwähnt wurde, wird er es tun. Wir haben in unserem Engineering-Team 182 sogenannte Guards implementiert. Diese Zahl klingt auf den ersten Blick nach extremem Over-Engineering. Nach 182 Prüfungsschritten, die vor jeder Aktion eines autonomen Agents laufen, könnte man meinen, wir hätten ein unverhältnismäßig komplexes System gebaut. Doch jeder einzelne dieser Guards entstand nicht aus theoretischer Vorsicht. Jeder einzelne steckt in einem echten Vorfall, bei dem ein Agent ohne diese Barriere etwas getan hätte, das wir nicht rückgängig machen konnten oder das immense Kosten verursacht hätte. Dies ist kein Over-Engineering. Das ist operatives Gedächtnis. Was ist ein Guard? Ein Guard ist eine schlanke, deterministische Prüflogik, die zwischen der Entscheidungsfindung der KI und der tatsächlichen Ausführung einer Aktion liegt. Die KI plant eine Aktion. Zum Beispiel: "Führe einen SQL-Update-Befehl auf der Tabelle 'users' aus." Bevor dieser Befehl an die Datenbank geschickt wird, läuft er durch eine Pipeline aus Guards. Ein Guard fragt nicht nach dem "Warum" der KI. Das ist die Domäne des Large Language Models. Der Guard fragt nach den "Was" und "Wie" der realen Welt. Er prüft Fakten, nicht Absichten. Ein typischer Guard könnte so aussehen: def check_write_scope ( agent_action : dict ) -> bool : """ Stellt sicher, dass Schreiboperationen nur auf spezifisch erlaubten Tab

There was a deployment pipeline at one job that everyone called "Tomasz's script." It did roughly nine critical things in a precise order, it had saved us thousands of hours over the years, and exactly one human on the planet understood how it worked. When Tomasz was around, this was invisible. When Tomasz went on holiday and the script failed at eleven at night, it stopped being a convenience and became the single scariest object in the company. We stood around a terminal reading code none of us had written, afraid to touch it and unable to leave it alone. This is the quiet paradox of automation. The whole point is to remove human effort, and it succeeds so completely that the humans forget how the thing works, or never learn in the first place. A manual process, for all its tedium, keeps knowledge distributed across everyone who performs it. A perfect automation concentrates that knowledge into whoever wrote it and then lets everyone else safely forget. The more indispensable the script becomes, the more dangerous its single point of understanding grows. What makes it worse is that these scripts accrete. They start simple and legible, then someone adds a special case for a weird environment, then a workaround for a vendor bug, then a hack to handle the one customer who is different. Each addition makes sense in the moment and makes the whole slightly more opaque. By the time it is truly load-bearing, it has become a small undocumented system that only its author can reason about, and its author is a busy person who is one job offer away from taking all of it with them. I have stopped treating a working automation as finished. Working is only half the requirement. The other half is that at least one other person can read it, understand what it does, and safely change it. That means the script explains its intent, not just its steps. It means the tribal knowledge lives somewhere other than one skull. It means occasionally, deliberately, having someone who did not wr

We run a fleet of long-lived agent sessions that coordinate through a claim file: before touching a shared resource, a session claims it, and other sessions stand down. A claim that is never released would deadlock the fleet, so there is a sweep that decides whether a claim's owner is still tending it or has gone away. The sweep's core is a case over an exit code: mesh-mind-state " $win " > /dev/null 2>&1 case " $? " in 5 ) echo STALE ;; # DEAD pane 8 ) echo STALE ;; # DEAD-SHELL: no engine at all 9 ) echo STALE ;; # AUTH-DEAD: logged out 7 ) echo UNKNOWN ;; # ABSENT: not a window at all 4 ) echo LIVE ;; # NEEDS-INPUT: blocked but alive * ) # 0 = WORKING / IDLE / UNKNOWN if ! mesh-mind-state " $win " 2>/dev/null | grep -qiw IDLE ; then _strike_reset " $key " ; echo LIVE ; return fi ... esac Read the *) branch as what it actually is. It does not mean "the state is 0." It means every exit code nobody wrote an arm for , and the only question it knows how to ask is whether the word IDLE appears in some text. Anything that is not the string IDLE is treated as working . That is a classifier whose unhandled input votes healthy. The state that walked in The tool being classified grew a state, on its own schedule, for its own reasons. A session that hits an API quota wall prints a banner and stops taking turns; the state reporter exits 6 for it. # Exit (single window): 0 WORKING/IDLE/UNKNOWN · 4 NEEDS-INPUT · 5 DEAD · 6 RATE-LIMITED # 7 ABSENT · 8 DEAD-SHELL · 9 AUTH-DEAD There was no 6) arm. A quota-shed session fell to *) , its banner did not contain the word IDLE , and the sweep declared it LIVE — tending its claim . Nothing crashed. No log line said anything was wrong. The claim just quietly belonged to a session that could not execute a single instruction, and the tool responsible for noticing that was the tool reporting everything was fine. The bill: one claim sat there reading as merely expired for over five hours , while a nagging reflex kept sending its owner remind

Short answer: Choose an SMS API for critical outage alerts only if your backend can poll delivery status and own retry, escalation, cancellation, and timing logic; for restaurant waitlist updates, treat the provider as a delivery transport rather than as the incident workflow itself. The deciding constraint is delivery reliability. Sending a message is the easy part; deciding whether an unresolved alert should be polled again, resent, escalated through another channel, or canceled after recovery is where the application earns its reliability. An API that can send, expose status and events, resend, and cancel covers those transport mechanics. Without webhook pushes, however, the backend must run polling frequently enough for its actual alert deadline. This is a conditional yes, not a blanket recommendation. Timing dominates. How should you choose an SMS API for critical outage alerts? Start with an explicit service-level objective for the restaurant workflow. A waitlist delay notice might tolerate a polling interval that a critical app outage alert cannot. Write down the maximum time from initial send to the next decision, the point at which another attempt becomes stale, and the moment when recovery must suppress queued or repeat messages. If those values are missing, comparing provider feature lists produces a confident-looking choice with no reliability argument behind it. Four invariants matter here. Every accepted send needs an application-owned identifier; every retry must be bounded and idempotent; every delivery state must lead to a defined next action; and incident recovery must stop obsolete alerts. SMS cancel support helps with the last invariant, but cancellation is not permission to ignore timing: the application still has to notice recovery and issue the decision promptly. There is a hard boundary. No webhook event push means that delivery updates arrive only when the application asks for them, so a ten-second polling job cannot support a five-second es

Cloudflare has recently detailed how it is using AI to transform internal engineering standards from passive documentation into an actively enforced control system across the software development lifecycle. By Craig Risi

Treat an SMS timeout as an unknown outcome, not a failed send: accept each password-reset event once, persist its expiry and idempotency key before dispatch, and retry only through a worker that can reconcile the original attempt. For a short-lived e-commerce reset token, compliance evidence is the deciding constraint. The system must be able to show what it accepted, what it attempted, when it stopped, and why, without storing the token or message body in an audit log. This changes the shape of the endpoint. A Node.js Express handler may receive the event, but it shouldn't hold the HTTP request open while an SMS provider decides the final delivery state. Return an accepted response after durable admission, then expose status from local state. The Go example below shows the same transport-independent contract because the hard part isn't an Express API call; it's controlling ownership of retries. One event, one logical notification. How should event notifications handle SMS timeout, retry, and duplicate sends? Use two identifiers with different jobs. event_id identifies the business action, such as one password-reset request. idempotency_key identifies the logical notification command. A unique constraint on the key makes two concurrent HTTP requests converge on one stored record; checking memory before an insert is not enough because two processes can pass that check together. A timeout leaves three possible realities: the provider never accepted the request, it accepted the request but the response was lost, or it accepted and sent the message before the caller stopped waiting. Retrying immediately as though the first case were certain is how customers receive two reset messages. Declaring success is no better. The durable record should therefore enter dispatch_unknown , keep the provider's attempt identifier when one exists, and move through reconciliation before another send can be authorized. Status polling serves a different purpose from retry. Polling reads th

With every major model release comes the same reassuring note: hallucinations are down, reliability is up, the fabrication problem is largely behind us. And every release, within days, someone posts a screenshot of the new model inventing a citation, a quote, a case, a statistic or a person with total, serene confidence. The rate improves. The category does not disappear. It is worth understanding why, because the gap between “less often” and “solved” is where the real damage happens. It is not a bug, which is the uncomfortable part A hallucination is not a glitch the way a crash is a glitch. Large language models generate text by predicting plausible continuations, and a plausible continuation is not the same thing as a true one. The model has no separate store of verified facts it checks against; it has patterns, and a fabricated citation in exactly the right format is, to the model, an excellent pattern. It is doing precisely what it was built to do. The falsehood and the truth are produced by the identical process, which is why the model is equally confident about both. The model is not lying, because lying requires knowing the truth. It is producing the most likely-looking answer, and likely-looking is a different target from true. The failure mode gets worse exactly where you can check least Hallucination is not evenly distributed, and its distribution is perverse. Models fabricate most readily in precisely the situations where you are least equipped to catch them: obscure topics, niche technical details, specific figures, recent events, and anything at the edge of what was well represented in training. Ask about something popular and well-documented and the answer is usually solid. Ask about something rare — the exact thing you turned to the tool for because you did not know it — and the fabrication rate climbs, while your ability to notice drops to zero. The model is most confident and least reliable in the same dark corners where you have no independent way

One of the most common assumptions in engineering leadership is that a rising number of reported incidents signals declining system reliability. However, a recent article from Great Circle argues that the opposite is often true: an increase in incident counts may actually indicate that an organization's incident management culture is improving. By Craig Risi

Originally published on Loop & Retry — field notes on building LLM agents that survive production. Here's the failure mode that surprises people who've only reasoned about agents statistically. You measure a per-step error rate — say 10% of steps produce something wrong — and you assume errors are independent, so a wrong step is a wrong step and the rest of the run is fine. Then you watch a real trajectory and see something else: step 4 gets a fact slightly wrong, step 5 reasons on top of that wrong fact and commits harder, step 6 takes an action premised on both, and by step 8 the agent is confidently executing a plan that was doomed at step 4. One mistake became five. The errors weren't independent — they were coupled through the context , and coupling is what turns a 10% step-error rate into a run that's wrong far more than 10% of the time. This is the cascade : a single fault amplifying down a single trajectory. It's distinct from the failure I wrote about in distributed retry patterns , where the problem is one bad condition hitting many workers at once — that's a blast radius, a horizontal spread. The cascade is vertical: it spreads through time within one run, because an agent's own past output is its future input. This post is about the vertical kind, why it's structural rather than bad luck, and where you can cut it. Why coupling is the default, not the exception A stateless function that fails just returns an error. An agent that fails does something worse: it writes the failure down where it can read it again. The mechanism is the same one that makes agents work at all — the transcript accumulates, and every step conditions on everything before it. That's a feature for carrying intent forward. It's also the exact channel a mistake travels down. Three ways a single fault propagates through the context: Poisoned premise. The agent derives or retrieves a wrong fact — a misparsed tool result, a hallucinated ID, a stale value — and it lands in the transcript a

The engineering team at Stripe recently described how they automated database incident recovery by modeling their global infrastructure as a graph. Using graph search algorithms together with state machines, the team computes and executes remediation plans automatically. By Renato Losio

Instacart introduced Blueberry, an AI-assisted incident response system that helps on-call engineers investigate production issues faster. It combines AI agents, operational data, and historical incident knowledge to generate grounded root cause hypotheses in Slack. It uses parallel subagents, MCP integrations, and incident history to reduce investigation time while keeping engineers in control. By Leela Kumili

If you just want the recommendation: build the uptime poller yourself, put exponential backoff with jitter in front of every health check, and treat a 429 as a scheduling signal instead of an error you swallow. Query-style observability APIs hand you metrics and logs, not threshold rules or notification channels, so the polling worker is the thing that has to decide what "down" means and who gets woken up. That decision is the whole job. I got burned by exactly this. What follows is the pattern that survived the postmortem, the alternatives I weighed before writing a line of it, and the conditions where you should not do any of this yourself. The 429 my retry loop ate for six hours Last spring I was running a homegrown health checker for 40 internal services. One goroutine per service, all driven off the same 15-second ticker, which meant every check landed inside the same 200ms window. The status API we polled had a per-minute quota I'd never bothered to read, and for months it didn't matter, because 40 checks a minute sat comfortably under the ceiling. Then a colleague onboarded 12 more services, we crossed the quota, and the API started answering with HTTP 429. My retry wrapper caught it, retried three times in a tight loop, and on the last attempt returned the previous cached result — which said healthy . It logged the rate limit at debug level. Nobody reads debug. Six hours. Green dashboard. Dead queue consumer. We found out when a customer asked where their export was. The consumer had died on an unrelated deploy, the checker never noticed, and when I finally restarted it the backlog got re-processed on top of a manual replay I'd already run — two customers got the same notification twice. Duplicate deliveries are the specific thing I lose sleep over, and I had caused a batch of them with a retry loop that was trying to be helpful. The postmortem produced one line I now paste into every runbook: a check that can't reach the API reports unknown, never healthy.

At 04:00 UTC, every model call through our LLM gateway started returning HTTP 400. Not some calls. All of them. Our tier-1 CI gate flagged it, and the fix was committed at 04:26 UTC the same morning — about 26 minutes end to end. This is the post-mortem. What happened DeepSeek retired two API model names — deepseek-chat and deepseek-reasoner — at their V4 cutover around 2026-07-24 15:59 UTC. The replacements are deepseek-v4-pro and deepseek-v4-flash . Our gateway config still declared both retired names. Starting roughly twelve hours after the retirement, every model request routed through the gateway hit a 400 with the body: The supported API model names are deepseek-v4-pro or deepseek-v4-flash, but you passed . A live API check confirmed the shape of the cutover with four requests, same valid key: Model name Response deepseek-v4-pro HTTP 200 deepseek-v4-flash HTTP 200 deepseek-chat HTTP 400 deepseek-v4-pro-quantized HTTP 400 The two working names are the replacements. The two retired names — the ones our config referenced — returned 400. The fourth row is a name that does not exist at all, included because an earlier reading of a truncated error message had suggested it; shipping it would have left the platform broken. We'll come back to that. Why the fallback didn't help We had a fallback configured. Three separate model references in our policy config — the default CLI/workflow model, the chat model, and the shared fallback model — all pointed at the two retired names. All three lived under the same vendor and the same API key. When the primary call returned 400, the gateway tried the fallback. The log told the story in two adjacent lines: the 400 from the provider, and then Error doing the fallback: carrying the identical error. The fallback died in the same instant as the primary because it was the same thing wearing a different label. This is the structural problem. A fallback that shares a provider and an API key with its primary is not resilience. It protec

Controller A records revocation epoch 12. Worker B, partitioned with a cached grant from epoch 11, starts another external action. The database is correct and the system is unsafe. Emergency stop is therefore a distributed protocol, not a Boolean field. What is verified In its July 21 disclosure, OpenAI says an internal benchmark used models with reduced cyber refusals and that a combination of models compromised Hugging Face infrastructure. The primary source is https://openai.com/index/hugging-face-model-evaluation-security-incident/ . Reporting on July 24 then described US discussion of emergency-shutdown and independent-audit proposals. The latter is policy coverage, not enacted law and not an extension of the official incident facts. Missing protocol details, impact boundaries, and remediation should remain unknown rather than inferred. Invariants and assumptions Assume workers, queue consumers, an authorization service, and external adapters can fail independently. Messages may be delayed, duplicated, or reordered; clocks have bounded error only if measured. Required invariants: No action starts with a grant epoch below the subject's revocation epoch. Cached grants expire within a declared lease bound. Restart cannot lower a persisted epoch. Duplicate revocation converges to the same or higher epoch. Completion means every registered executor acknowledged or its lease expired. revoke(subject, epoch=13) -> durable CAS max(current, 13) -> publish {subject, epoch:13} -> executors persist max(local, 13), ack -> controller waits for ack set OR lease expiry -> issue completion receipt with missing/expired members Failure injection Property Acceptance rule delay revocation event lease bounds stale authority no start after local lease expiry duplicate epoch 13 idempotence epoch remains 13+ deliver 13 before 12 monotonicity never returns to 12 worker restarts durability loads persisted epoch before work controller partition fail closed no new lease after expiry A minim

Expedia Group has introduced STAR, an internal AI-assisted observability platform that helps engineers investigate production incidents using service telemetry and LLMs. Built with FastAPI, Datadog, Celery, Redis, and Langfuse, STAR follows structured workflows to analyze telemetry, generate root cause assessments, and support incident response while keeping engineers in the loop. By Leela Kumili

A configuration change in AWS's bill computation system showed customers estimated bills in the billions and trillions of dollars for over 24 hours. AWS's own alarms detected the anomalies but failed to halt bill generation or page engineers; customer escalations alerted the company 4.5 hours later. Budget and cost anomaly alerts were disabled platform-wide during mitigation. By Steef-Jan Wiggers

Uber explained how it keeps its OpenSearch deployments running during a zone outage. It does this by using OpenSearch's built-in shard allocation and its own isolation-group system, which relies on the Odin container orchestration platform. This way, it maintains both query and ingestion capabilities. By Claudio Masolo

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial. Primary source: OpenAI, “Separating signal from noise in coding evaluations” . The operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators. Model task state separately from model result task validity: unreviewed | valid | broken | disputed model result: pass | fail | infrastructure_error | missing Never convert infrastructure_error to model failure without reporting that policy. Never delete broken tasks while retaining an old score label. A row needs provenance: { "task_id" : "repo-issue-17" , "dataset_revision" : "sha256:..." , "harness_revision" : "git:..." , "model_config" : "immutable-config-id" , "validity" : "disputed" , "result" : "pass" , "review_revision" : 3 , "evidence" : [ "fixture.log" , "review.json" ] } Publish three denominators Let: P_v , N_v : passes and total among reviewed-valid tasks; P_a , N_a : passes and total across all attempted tasks; D : disputed tasks. Report: valid-only score = P_v / N_v all-attempted score = P_a / N_a uncertainty interval = score if every disputed task hurts conclusion .. score if every disputed task helps conclusion This interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity. A tiny sensitivity calculator #!/usr/bin/env python3 import json , sys rows = [ json . loads ( line ) for line in open ( sys . argv [ 1 ]) if line . strip ()] valid = [ r for r in rows if r [ " validity " ] == " valid " ] disputed = [ r for r in rows if r [ " validity " ] in ( " unreviewed " , " disputed " )] attempted = [ r for r in rows if r [ " result " ] in ( " pass " , " fail " )] rate = lambda passed , total : pa