标签：#bug

The number that lied Friday May 15, 4:13 PM. The Sentry alert pings on my phone. The first Phase 1 re-enrolling student waits in front of the payment screen, her name at the top of my tab. I put down the can, I reopen the screen. The mug with Françoise's face on it, on the desk next door, catches a yellow reflection I notice without looking at. The stack trace fills the screen. The stack trace opens, nine fields out of ten at null , and a number I didn't see coming. type = "StripeConnectionError" message = "An error occurred with our connection to Stripe." code = null statusCode = null requestId = null duration = 9 ms Nine milliseconds. On a Vercel route in Paris region, DNS resolves in forty ms, a TLS handshake costs one to two hundred. Nine milliseconds isn't a network call that failed. It's a network call that never happened. The SDK didn't reach the wire. Instinct immediately offers three patches. Vercel serverless timeout — I add maxDuration , redeploy. Revoked key — I'll rotate it. Stripe account restricted after the live switch — I open a support ticket. These three hypotheses are plausible. None of the three is falsifiable from the symptom alone, and that's precisely what makes them dangerous: each opens a fifteen-to-thirty-minute cycle with rollback at the end if it's wrong. Multiplied by three, half a day lost with the customer still clicking. I don't have time. A student is waiting. Four tests, in order I know the incident class — "preview works, prod breaks" , or its mirror. The rule for this class is that you fix nothing until you've discriminated the layers. Four tests, executed in order. Each eliminates a family of hypotheses, not an isolated hypothesis. And each is designed to refute what it interrogates — because a test that seeks to confirm always finds, by selection, what it's looking for. Test 1 — reproduce in the witness environment. I rerun the same funnel in preview, with the sk_test_ key. Checkout opens in three hundred fourteen milliseconds,

It's 2:47am. The pager has just gone off for the third time in twenty minutes. Checkout latency is spiking. The error rate on /api/orders is climbing. Slack is filling with screenshots of half-finished trace views. Somewhere in your logs, the answer is sitting there in plain text, buried under a few million other lines that all look just as urgent. This is the moment people are talking about when they say "AI is going to change how we debug production." Not the demo where someone asks ChatGPT to write a regex. The 2:47am moment. The one where a tired human has to hold five tabs open in their head and form a hypothesis before the executive team starts asking for an ETA. It turns out that's where the technology has the most to offer, and also where it embarrasses itself most often. Let's break down what's actually working in 2026, where the seams still show, and how to wire an LLM into your incident-response loop so it earns its keep instead of just adding another window to glance at. What AI is genuinely good at during an incident The two boring superpowers first: reading fast and correlating across heterogeneous signals . Those are the things humans get worst at when they're tired and time-pressured, and they're the things a good LLM does at the same speed at 2am as at 2pm. Datadog's Bits AI SRE, which the company benchmarked against real incidents from hundreds of internal Datadog teams, is built around exactly this insight: an agent that can fan out across metrics, logs, traces, recent deploys, and incident history simultaneously, then collapse the findings into a single readable narrative. Datadog runs the agent against tens of thousands of evaluation scenarios and claims time-to-resolution wins of up to 95% in its published material. That headline number is marketing (you should always read it as "in the cases where the agent worked, this is what it shaved"), but the underlying capability is real, and it isn't unique to Datadog. Honeycomb's Query Assistant has b

A receiver pulling a UDP feed was missing roughly 30% of its messages. No errors, no exceptions, no stack traces — just gaps in the sequence numbers. The first suspect is always the network: a flaky switch, a saturated link, a tired NIC. The network was innocent. The packets were being dropped on the receiving host , after they'd already arrived. Here's how to tell the difference, and why it matters. Why UDP makes this sneaky UDP has no retransmission and no backpressure. When a datagram is lost, nobody is notified — not the sender, not the receiver. The packet simply isn't there. That means two completely different failures look identical from the application's point of view: The network dropped the packet before it reached your machine. Your own host accepted the packet and then threw it away after it arrived. The application sees the same thing in both cases: a missing sequence number. But the fix is in a different building depending on which one it is. Where the packets actually go The receive path is: NIC → kernel socket receive buffer → your recv() call. The kernel parks incoming datagrams in a per-socket buffer until your code reads them. If your code doesn't drain that buffer fast enough, it fills, and the kernel drops the overflow. Crucially, the kernel counts those drops. On Linux: # Per-protocol summary — look for "receive buffer errors" netstat -su # Or straight from the kernel counters cat /proc/net/snmp | grep -A1 Udp # InDatagrams ... InErrors RcvbufErrors ... If RcvbufErrors is climbing, the network did its job and your host discarded the datagrams. That single counter collapses a week of "is it the switch?" into about ten seconds of certainty. The actual cause In this case the socket receive buffer was sitting at the default (~208 KB). The sender burst faster than a single receive thread could call recv() . Average throughput looked fine on every dashboard — but the bursts filled the buffer in milliseconds, and everything past the brim was dropped.

Your CI was green last Friday. Today, the payments test is failing. Somewhere between Friday's merge and now, 47 commits landed on main . Which one broke it? Most developers answer this the wrong way: they scroll through git log , check out suspicious commits one by one, and run the test manually. An hour later, they're still guessing. There's a command built for this exact problem. It's called git bisect , and once you learn it, you'll never debug regressions the old way again. How bisect works git bisect is a binary search across your commit history. You tell Git two things: A good commit (a known point where the bug didn't exist) A bad commit (a known point where the bug exists — usually HEAD ) Git then checks out the commit halfway between them. You test. You mark it as good or bad. Git narrows the range by half. Repeat. With 47 commits between "good" and "bad", it takes at most 6 steps (log₂ 47) to find the exact commit that introduced the bug. Versus checking every commit manually, that's the difference between 5 minutes and an hour. The manual workflow # Start a bisect session $ git bisect start # Mark the current state (HEAD) as bad $ git bisect bad # Mark a known-good commit $ git bisect good a3f1d22 # Bisecting: 23 revisions left to test after this (roughly 5 steps) # [7e4b9c1] refactor: extract payment validator # Git has checked out a commit in the middle. Run your test. $ npm test -- --grep "payments" # Test passed — mark this commit as good $ git bisect good # Bisecting: 11 revisions left to test after this (roughly 4 steps) # [b2d8e11] feat: add retry logic to payment API # Test failed — mark this commit as bad $ git bisect bad # ... continue until Git announces the first bad commit: # b2d8e11 is the first bad commit # commit b2d8e11 # Author: leo@company.com # Date: Tue Apr 15 11:42:03 # feat: add retry logic to payment API # Done — reset to where you started $ git bisect reset In 6 commands, you know exactly which commit broke the tests. No guessing

A user picks a video from Photos. The app turns that video into a file the server can accept. Then it uploads the file. On the surface, this sounds like a simple upload flow. In practice, iOS makes you answer a much more specific question: How do you reliably turn a video from the user's Photo Library into an uploadable MP4 file? At first, I used AVAssetExportPresetPassthrough . It looked like the right default. It preserves the original media as much as possible, avoids unnecessary re-encoding, and is fast when it works. No quality loss, less CPU usage, less battery cost. But some videos failed during export. The error was usually in the AVFoundationErrorDomain Code=-11838 family. Apple describes this error as operationNotSupportedForAsset : an operation was attempted that is not supported for the asset. At first, I suspected an iCloud download issue, a Photos permission edge case, or something retryable inside PHImageManager . That was not the real problem. The actual problem was more fundamental: I was trying to write an iPhone MOV asset into an MP4 file using a passthrough export. The Original Flow The original code was roughly shaped like this: PHImageManager . default () . requestExportSession ( forVideo : asset , options : options , exportPreset : AVAssetExportPresetPassthrough ) { exportSession , info in exportSession ? . outputURL = outputURL exportSession ? . outputFileType = . mp4 exportSession ? . exportAsynchronously { // upload } } The intention was reasonable: Ask Photos for an export session for the PHAsset . Use AVAssetExportPresetPassthrough to preserve the original quality. Write the result as an .mp4 file for upload. Many videos exported successfully this way. That is what made the bug confusing. Some videos worked. Some did not. The hidden assumption was: Any iPhone video can become an MP4 file through a passthrough export. That assumption is not always true. iPhone Videos Are Usually MOV Files To a user, it is just "a video." Internally, an iPh

Skills Are a Mess. Let's Fix That. Here's the problem: you write a skill for zeroclaw. It works locally. You push it. Someone else tries to install it. Nothing works. The error says "missing dependency" but doesn't say which one. Or it installs but the audit fails silently. Or the test harness just... doesn't run. Sound familiar? I've been watching the zeroclaw skills ecosystem grow. More people are authoring skills. More people are hitting the same walls. The v0.7.6 release is about tearing those walls down. What Actually Breaks Let's get specific. Three failure modes I see every week: 1. Install hell zeroclaw skills install my-skill # → Error: Failed to resolve dependency graph # → (no other output) You're left guessing. Is it a peer dependency conflict? A missing Python version? A circular reference in the skill manifest? The loader gives you nothing. 2. Audit blindness zeroclaw skills audit ./my-skill # → Audit complete. 0 issues found. # → (skill crashes immediately on first use) The audit passed. But it didn't check for the actual runtime errors — missing environment variables, incompatible tool signatures, malformed output schemas. It checked the manifest format. That's it. 3. Test harness that doesn't test zeroclaw skills test ./my-skill # → Running 3 test cases... # → All passed. # → (skill still hallucinates in production) The test harness runs your skill against mock data. But the mock data doesn't match real tool outputs. Your skill passes locally, fails in the wild. Why This Happens The current architecture treats skills as static packages. You define metadata in a skill.json , point to some functions, and assume it works. But skills are dynamic. They call tools. They depend on runtime state. They interact with the sandbox. The loader doesn't validate the runtime contract. The audit doesn't simulate execution. The test harness doesn't fuzz inputs. So you get false positives everywhere. "Works on my machine" becomes "works in my specific environment with

The RL Flywheel That Actually Works Here's what's breaking: You've got a reinforcement learning setup that trains, validates, deploys, and then... nothing. No feedback loop. No automatic retraining. No safety gates. Just a model that gets stale the moment it hits production. Sound familiar? I've been building RL systems for a decade. The pattern is always the same: great training pipeline, terrible deployment loop. You spend weeks getting 95% validation accuracy, push to prod, and three days later the distribution shifts. Your agent starts making garbage decisions. You scramble to retrain. Rinse. Repeat. This sucks. I know. The Real Problem The issue isn't training. It's the feedback loop . Most RL systems have: Training pipeline — works fine Validation — mostly works Deployment — fire and forget Observation — maybe some metrics Strategy update — manual, if ever Step 4 and 5 are broken. You're flying blind after deployment. The Flywheel Architecture Here's what a real RL flywheel looks like: Train → Simulate → Validate → Gate → Deploy → Observe → Analyze → Train Every arrow is automated. Every gate is a hard check. Every observation feeds back into training strategy. Let me show you the actual implementation. The Training Loop class RLFlywheel : def __init__ ( self ): self . model = Model () self . buffer = ReplayBuffer ( 1_000_000 ) self . safety_gate = SafetyGate () self . observer = OnlineObserver () def train_epoch ( self , episodes = 1000 ): for ep in range ( episodes ): states , actions , rewards = self . simulate_episode () self . buffer . store ( states , actions , rewards ) batch = self . buffer . sample ( 256 ) loss = self . model . update ( batch ) # Validate against known failure modes validation_score = self . safety_gate . validate ( self . model ) return loss , validation_score Notice the validation happens during training, not after. That's the first gate. The Safety Gate class SafetyGate : def __init__ ( self , thresholds ): self . thresholds = thre

An intentionally vulnerable e-commerce platform that teaches you to find, exploit, and understand IDOR vulnerabilities — the way they actually appear in the wild. Let's talk about the most deceptively simple vulnerability in web security: IDOR . On paper, it sounds trivial — change a number in the URL, access someone else's data, collect your bounty. But anyone who's spent real time hunting knows the truth: IDORs in production applications are rarely that obvious. They hide in request bodies, lurk inside multi-step workflows, and disguise themselves behind modern frontend frameworks that abstract away the very IDs you're supposed to manipulate. That gap — between textbook IDOR and real-world IDOR — is exactly where IDOR BugBounty Labs lives. What Is IDOR BugBounty Labs? It's an open-source, Node.js/Express e-commerce application built with one purpose: to give you a realistic playground for practicing IDOR attacks. Not simulated. Not theoretical. Intentionally vulnerable, locally hosted, and designed to mirror the complexity of actual Bug Bounty targets. Built with Express and TailwindCSS, it simulates a functioning online store — complete with user accounts, orders, addresses, support tickets, notification settings, and a checkout flow. Every feature contains at least one authorization flaw waiting to be exploited. Why This Lab Is Different Most IDOR labs give you one obvious URL parameter to change and call it a day. This one doesn't. IDOR BugBounty Labs includes: 5 distinct challenges ranging from easy to hard 3 different IDOR types: URL parameters, request bodies, and hidden body parameters Both read and write IDORs — accessing data and modifying it Multi-step business logic that mimics real e-commerce flows A flag submission system so you can verify your findings The challenges don't just teach you to change an ID. They teach you to think about where IDs live, how they're passed, and what happens when authorization checks are missing. The 5 Challenges 1. Read O

A public spat between Microsoft and an independent security researcher reopens a long-running debate over who is responsible for securing software.

The Transition A few years ago, bug hunting was a manual craft. You scanned subdomains with one tool, tested endpoints with another, and stitched results together by hand. Today, AI changes the speed entirely. Not by replacing the hunter. By eliminating the boring parts. What AI Actually Changes 1. Reconnaissance at Scale Subdomain enumeration, port scanning, and technology fingerprinting used to take hours. AI-powered pipelines now do this in minutes: Passive reconnaissance via Certificate Transparency logs, search engines, and DNS records Automated crawling and endpoint discovery Technology stack detection from response headers and HTML patterns JavaScript file analysis for hidden endpoints and API keys The machine does the grunt work. The human interprets the results. 2. Pattern Recognition Vulnerability classes have signatures. SQL injection looks different from XSS, which looks different from SSRF. AI models trained on thousands of real vulnerabilities can flag suspicious patterns faster than manual code review. This is not about finding zero-days. It is about catching the low-hanging fruit that everyone else misses because they are in a hurry. 3. Intelligent Fuzzing Traditional fuzzers throw random data at endpoints and wait for crashes. AI-guided fuzzers understand the input format and generate test cases that explore edge cases a human would not think of. The result: fewer requests, better coverage, higher signal-to-noise ratio. Where AI Struggles Business Logic Flaws AI does not understand your application's purpose. It cannot tell if a discount code is applied twice, or if a user can access another user's private data through a convoluted API flow. These are the vulnerabilities that require context. Human context. Authentication Logic Authentication bypasses are often creative. They exploit the gap between what the developer intended and what the code actually enforces. AI can find simple auth flaws, but multi-step authentication bypass chains still need h

We're updating our bug bounty program standards to prioritize quality submissions, clarify shared responsibility boundaries, and evolve how we reward low-risk findings. The post Raising the bar: Quality, shared responsibility, and the future of GitHub’s bug bounty program appeared first on The GitHub Blog .