标签：#Debugging

I had a table in the database that was supposed to fill itself. Its job was to learn from failures : every time the system tried a variant of something and it didn't work, it saved it, to recycle later in another context where it might. A laboratory of failed attempts, piling up. In production it had zero rows . It had been deployed for days and hadn't saved a single record. Meanwhile a neighbouring table —another memory, the one that notes which work is already exhausted so as not to repeat it— was growing normally. The temptation is obvious: there's a bug in the write. I went looking for it, and it wasn't there. Zero rows isn't the same as a write error The path that saves into that table emits a warning if the write fails. I searched the logs for those warnings: zero . No write had failed. That's a fact, not an absence of one. If the path had been taken and had failed, it would have left a trace. Zero traces and zero rows fit only one explanation: the write path never ran . Not ran-and-failed. Didn't run. That's the difference between a real negative and a negative that was never put to the test, and they look the same unless you look for the positive control —something the log WOULD show if the path had been taken—. Without it, "healthy and quiet" and "dead" look identical. Two mechanisms starving each other Why didn't it run? Because of another mechanism, upstream, doing its job well. That system has a negative memory : when it exhausts everything it knows how to try against a target, it notes it down, so as not to spend effort again on something it already knows won't pay. It's a sensible optimisation. But it sat before the phase that generated new variants —the phase that, on failing, would have fed the library—. As soon as a target went "exhausted", that phase was skipped entirely . And if the phase never runs, it never produces a failure to save. Each mechanism, on its own, is correct. The negative memory avoids useless work. The library learns from failure

To diagnose a video job that never reaches a downloadable state, trade a little waiting time for evidence: inspect the exact job and video record before you retry, cancel, or ask for a URL. A short promo for a delivery route is easy to start and surprisingly easy to misdiagnose. A download request is the last step, not a health check. Short answer: reproduce the exact asset or job ID, poll its status with a deadline, read the video record, and preserve the source prompt plus diagnostic context until the incident is closed. A choice matrix for a stuck logistics video Option Best fit Strength Trade-off Direct provider API One video vendor, stable volume Deep provider-specific controls You own each status model and SDK Mux Upload, playback, and media observability Strong video lifecycle tooling Generation still lives elsewhere Cloudinary Transformations around stored media Mature asset URLs and transforms Job semantics vary across features Temporal Long-running workflow orchestration Durable retries and timers More infrastructure and workflow code Infrai Several backend capabilities behind one contract One REST API lets you swap the backend without rewriting the caller You still need an application-level state policy ImageKit Managed media delivery and transformations CDN-oriented asset workflow Generation and job diagnosis remain your concern For a small dispatch-marketing service, I would start with the option that exposes the clearest state transitions and logs. Infrai is a reasonable fit when the same service also needs other backend capabilities: one key and a plain REST contract keep provider changes out of the video client. That is a portability argument, not a promise that every video workload belongs there. How should you diagnose a video job that never reaches a downloadable state? Start with identity. Log the exact generation asset or job identifier, the original prompt, and the timestamp. If a retry creates a second job before you have captured that context

Author's Note / Disclosure: 100% human-authored content based on real production engineering work. No AI was involved in writing the article, technical analysis, or code. cy.session() is the single biggest speed win available to an authenticated Cypress suite. You log in once, Cypress snapshots cookies, localStorage and sessionStorage , and every later spec restores that snapshot instead of walking through an identity provider. The safety net is validate() . Cypress runs it after restoring a cached session; if it throws, fails an assertion, or yields false , Cypress throws the snapshot away and runs setup again. That is the whole contract: a bad session gets detected and replaced. Mine could not fail. For weeks. And it cost me days of chasing "flaky" specs that were nothing of the kind. The code that looked fine Cypress . Commands . add ( ' login ' , ( user : User ) => { cy . session ( user . username , () => { cy . visit ( ' / ' ) cy . origin ( idpOrigin , { args : user }, ({ username , password }) => { cy . get ( ' #username ' ). type ( username ) cy . get ( ' #password ' ). type ( password , { log : false }) cy . get ( ' button[type="submit"] ' ). click () }) cy . get ( ' #app-shell ' ). should ( ' be.visible ' ) }, { cacheAcrossSpecs : true , validate () { cy . request ( ' /connect/userinfo ' ). its ( ' status ' ). should ( ' eq ' , 200 ) }, }, ) }) Reasonable, right? /connect/userinfo is the OIDC user info endpoint. If the session is dead it should 401, validate() fails, and we log in again. Why it always passes Two independent bugs stack up here, and either one alone is enough to make the check worthless. The URL is relative. cy.request('/connect/userinfo') resolves against baseUrl , which is the application, not the identity provider. So the request never touches the IdP. The application is a single-page app. Its host serves index.html for any path it does not recognise, because that is what history-API routing requires. A request for /connect/userinfo gets b

Payment reconciliation can look completely reliable until a calendar edge case exposes a hidden...

A pull request arrived after midnight with a README that claimed the API was already healthy. The coding agent had started a process, requested its own localhost, and treated a 200 as proof the service would run for everyone. That response was genuine inside a short-lived workspace, yet it said nothing about the laptop waiting on Monday. The reviewer stared at a green sentence printed on a host that nobody on the team could reopen. This pattern appears whenever a coding agent can execute commands, not merely suggest them, and reviewers misread the transcript. Developers treat the agent's shell as a preview of their laptop because both sessions speak bash and render similar fonts. The analogy fails like a hotel gym standing in for a home garage, familiar until one bolt size changes. Claims in the next sections are the ones that keep returning during review, then a fingerprint workflow that makes the rental visible. Myth: a bound port means the service is portable Agents love a bound port because it is a crisp success token that copies cleanly into a README. A process that answers on the sandbox does not encode libc, extra packages, file layout, or the user's group permissions. Health checks measure a moment on a host you do not retain, not a contract with the checkout that will survive merge. Treat a remote 200 as proof that some files ran once, then demand a second run on CI or a laptop. A useful correction is to refuse README claims that cannot be replayed from a clean clone of the branch. Ask the agent for the exact command sequence, the working directory, and the non-secret environment keys it exported during the run. Then execute that sequence locally with undocumented keys unset, unless they already exist in the team's dotenv template. If the local run dies on a missing header or a path the sandbox invented, the original green check was a rental. Myth: a free remote box is unofficial CI Teams under schedule pressure will point at agent logs the way they once po

A program of mine hung for forty minutes. Not spinning at a thousand loops a second: at zero percent CPU . It wasn't doing too much work; it wasn't doing any work at all. And still it wouldn't finish. The program does something common: it orchestrates external command-line tools. It launches one, reads what it writes to standard output, and moves on to the next when it's done. So it doesn't get stuck when a tool drags, each one has a timeout: when it fires, the process is killed and we carry on. That's the part that failed, and it failed where no one looks: after killing the process. Killing the process doesn't close the pipe When you read a subprocess's output, you read from a pipe : one end writes (the subprocess), the other reads (you). Your reader doesn't finish when the subprocess dies. It finishes when EOF arrives, and a pipe's EOF arrives only when the last write end is closed. Almost always they coincide: the subprocess is the only writer, it dies, its end closes, EOF arrives, your reader finishes. All in microseconds. But "almost always" isn't "always". The tool I launched launched another one in turn —a grandchild—. And that grandchild inherited the pipe's write end, because on Unix a child inherits its parent's open descriptors unless told otherwise. So when the timeout fired, I killed the child. Its end closed. But the grandchild was still alive , with its copy of the descriptor open. The last write end hadn't closed. EOF never came. And my reader sat waiting for an EOF that would never arrive —at zero percent CPU, blocked in a read() , indistinguishable from slow work—. The symptom that deceives What makes this failure so hard to see is that it doesn't look like a failure . An infinite-loop hang burns CPU: you see it in top instantly. This one spends nothing. The thread is asleep in the kernel waiting for data that isn't coming. In the process list it looks healthy. In the metrics it looks like it's "taking a while". The only way to tell "hung forever"

A random game is easier to debug when the same inputs produce the same result. In HoopTrait, a browser basketball project, the Lab mode combines eight selected traits and generates a fictional career. The interesting engineering problem is keeping replay, sharing and validation consistent. This is a technical development note, not a claim that a game score predicts an athlete's real performance. Store the decisions, not just the result The Lab state records a seed, a dataset version and an ordered list of actions. An action is a pick or a reroll. Replaying those actions reconstructs the build. A seed alone is not a complete replay contract: changing the player pool or its order can change a seeded draw. A dataset version therefore matters alongside the random seed. For a future release, the same principle should apply to changes in the rules themselves. Test invariants across many runs The Lab test suite iterates through 1,000 seeds. For each seed it shuffles the order of the eight skills, uses the two allowed rerolls, and completes a build. It checks that: Eight distinct players were selected. All eight traits are present, and no player remains to be drawn after completion. The overall game score stays between 0 and 99 and matches the shared rating function. Packing and unpacking the share state returns the original state. Recomputing the fictional career returns the same output. The ten simulated seasons sum to the displayed career earnings. Those assertions catch different problems. A stable score does not prove that a shared link reproduces the same selections. A complete build does not prove that its season totals add up. Reject impossible histories A share payload is untrusted input, even in a client-side game. Negative or fractional seeds, duplicate skill picks, a third reroll, unknown action types, a mismatched dataset version and actions after completion are rejected. The tests also cover malformed encoded payloads and unexpected fields. Local state is usef

When adb devices does not show the result you expect, reinstalling random drivers is rarely the best first move. The output already tells you which layer is failing. This checklist separates the most common states: device unauthorized offline An empty device list ADB not recognized by the terminal The goal is to diagnose the connection in a logical order: tool, cable, USB mode, authorization, and finally drivers. Before troubleshooting Make sure the basic setup is correct: Install the latest Android SDK Platform-Tools from Google. Use a USB cable that supports data, not only charging. Unlock the Android phone. Enable Developer options and USB debugging. Connect directly to the computer when possible instead of using an unpowered hub. The location of Developer options differs between Samsung, Xiaomi, Pixel, Huawei, OnePlus, and other interfaces. If you need the device-specific menu paths, this guide to enabling USB debugging on Android phones covers the common manufacturers and the RSA authorization step. Start with one command Open Terminal, PowerShell, or Command Prompt inside the Platform-Tools folder and run: adb devices For extra information, use: adb devices -l A normal result looks similar to this: List of devices attached R58M123ABCD device product:example model:Example device:example The word after the serial number is the important part. What each ADB state means Result Meaning Where to look first device ADB can communicate with the phone The connection is ready unauthorized The phone has not authorized this computer Phone screen and RSA prompt offline ADB sees the device but cannot communicate reliably ADB server, cable, port, or device Empty list The computer is not exposing the phone to ADB Cable, USB mode, driver, or debugging setting adb not recognized The shell cannot find the ADB executable Platform-Tools folder or PATH Case 1: The result is device This is the success state. ADB can send commands to the phone. You can test the connection with a harml

Our app ships long video lessons. Twenty to forty minutes each, watched mostly on hostel wifi and mobile data with two bars. Streaming video looks like a solved problem when you build the happy path. You wire up ExoPlayer, point it at a stream, it plays. Then real students open it on devices you have never held, and your bug tracker starts filling up with one line: "video not playing." That line turned out to be four unrelated problems. Here's what I actually learned. ## Buffering is the bug you feel before you see an error Nobody waits out a bad connection during a 35-minute lecture. On a 15-second reel a stall is annoying. In a lecture, a stall every two minutes means the student closes the app and studies from a PDF instead. So the real question was never "does it play." It was "what happens when the network stops cooperating for four seconds." Three situations broke us repeatedly: Wifi to mobile data handover mid-lesson Bandwidth that is technically connected but useless Short total drops, three to ten seconds, that should not end playback ExoPlayer gives you everything you need here. Adaptive bitrate, a configurable LoadControl , player listeners. The catch is that the defaults are tuned for general media, not for a 40-minute lecture on a bad link. What moved the numbers for us: Bigger buffers, deliberately. DefaultLoadControl.Builder().setBufferDurationsMs(...) takes a min buffer, a max buffer, the buffer needed to start playback, and the buffer needed to resume after a rebuffer. That last parameter is the interesting one. Raising it costs you a little extra time on resume and buys you far fewer repeat stalls, because the player stops trying to restart on a nearly empty buffer. Telling the user the truth. Our first version showed a spinner and nothing else. A spinner with no context reads as "the app is frozen," so students force-closed and reopened, which threw away the buffer and made everything worse. Saying "reconnecting" instead of spinning silently cut t

Have you ever watched a two-kilobyte write fail with No space left on device while df -h still showed free gigabytes? I did, and I spent the next forty-eight hours cleaning the wrong evidence. This is the reconstructed field notebook from that session, including the commands I ran, the ones that misled me, and the checklist I now run before I blame the disk. Nothing here is a benchmark, a quota promise, or a claim about hardware I did not measure. I was iterating on a small Python worker that dumped JSON sidecars next to each run. The worker itself was unremarkable. The failure mode was not. Hour 0: the write that should have been boring The first traceback looked like a disk problem, so I treated it like a disk problem. Would you have done anything else with ENOSPC staring at you from a three-line stack? I would not, and that is exactly how the next two days started. OSError: [Errno 28] No space left on device: 'runs/2026-09-05T07-12-04.json' I ran the obvious command, got a comforting number, and closed the wrong investigation. df -h reported plenty of space on the root filesystem, and /tmp looked equally relaxed. I even created a dummy file in $HOME by hand, which succeeded, so I told myself the worker path was special. df -h df -h /tmp /var /home touch ~/probe-ok.txt && ls -l ~/probe-ok.txt That last touch was the trap. Can a filesystem accept a file in one directory and refuse a tiny file in another while still having blocks to spare? Yes, and inode exhaustion is the boring reason. I did not ask that question for twelve hours. What I tried first, and why it felt reasonable I treated the symptom as log rot, because that is the story operators tell each other. I truncated worker logs, deleted old JSON sidecars I could see, and reran the job with a smaller batch. The write still failed, sometimes on file number twenty, sometimes on file number four. Truncated worker.log and debug.log with : > file instead of deleting the path. Removed a handful of large .jsonl fil

I was asked whether the system was sending patient data to an external body while the integration was half-built. I went and read the logs of every run. They all died early: some with a 415 because the content type wasn't what the other end expected, others with a 500. Not one showed an outbound call. I answered that nothing was going out. The first run that got past the 500 sent two requests carrying real clinical data . My answer had been false from the start, and the worst part is that it was false in a way that felt rigorous: I had looked. I had evidence. The evidence was logs of real executions, not assumptions. A negative says nothing on its own The mistake wasn't misreading the logs. It was not noticing what produced that silence. The runs died before reaching the code that sends. The log didn't say "I didn't send"; it said "I never got to the part that sends". Those are two different statements and they produce exactly the same output: nothing. That's the general shape of the problem, and it turns up everywhere once you look for it: A counter at zero can mean "it didn't happen" or "the counter was never incremented". A "not found" can mean "it doesn't exist" or "I looked in the wrong place". A green test can mean "it passed" or "it skipped itself". An exit 0 can mean "it worked" or "the command was strangled by a pipe that swallowed the exit code". A silent dashboard can mean "everything is fine" or "the process feeding it has been dead for three weeks". In all five, the evidence is identical. And in all five, the optimistic reading is the reassuring one, so it's the one chosen without thinking. The positive control The fix isn't to be more suspicious. It's to demand one specific thing before accepting any negative: Find something the log MUST show if the path was actually taken. If the system had reached the part that sends, something would have to appear in the log: the "preparing request" line, the batch identifier, the connection attempt. Any signal that

A production postmortem. We shipped browser-side image compression (Rust → WASM + WebGPU), turned on cross-origin isolation for speed, and watched every format crash with compression worker crashed . Here's the root cause and the fix. The setup We built an image compressor that runs 100% in the browser — Rust compiled to WASM for the codec work, WebGPU for the heavy ML passes (background removal, denoise, watermark). No upload, so users' pixels never leave the device. Privacy is the whole selling point. For the multi-threaded code paths we rely on shared memory + atomics , which in the browser requires crossOriginIsolated . So we served the document with: Cross-Origin-Embedder-Policy: require-corp Cross-Origin-Opener-Policy: same-origin That gives us crossOriginIsolated === true , unlocks SharedArrayBuffer , and lets the *‑threaded WASM builds actually spawn workers. The build uses a nightly toolchain ( nightly-2025-06-01 + -Z build-std ) with: RUSTFLAGS = "--cfg=... +atomics,+bulk-memory --shared-memory --import-memory" and a custom rayon handle pool ( with_turbo_pool ) instead of build_global , so we control worker lifecycle and can abort/self-heal. The incident After flipping COEP to require-corp in production, every format started crashing with the same message: compression worker crashed Not one codec — JPG, PNG, WebP, AVIF, all of them. It was a P0: the core feature was dead for every user. What made it nasty: it only reproduced under real cross-origin isolation . Local dev without COEP was fine. Staging without the header was fine. So the bug hid until it hit production traffic. Root cause The *‑threaded WASM packages spin up nested rayon workers to parallelize the codec. Under COI + COEP require-corp , those nested workers get blocked by Cross-Origin-Resource-Policy / COEP — the spawned worker script is treated as a cross-origin response without the right CORP header, so the browser refuses it. No worker → the rayon pool never initializes → the compression c

The happy path was never the bottleneck. I was timing successes and shipping a miss. Production traffic is full of misses. Would you trust a bench that never fails? An AI rewrite loves the clean try. It wraps a lookup in except KeyError. It logs the miss "for observability." It looks professional. It is also a tiny furnace. Exceptions are not cheap branches. Log formatters are not free either. I learned that the loud way. Cheap generation makes the trap faster. A model will emit a polite miss path before you blink. Technical debt used to wait for a human. Now it arrives as a helpful patch tonight. The debt is not the lookup. The debt is a story about speed with no miss mix in the graph. I needed variants, not vibes. I used MonkeyCode's free model access and free server option to draft those variants. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model proposes shapes. It does not know your miss rate. If the graph disagrees, the patch dies. The lab I actually rerun This is a pocket harness. It is not a production claim. Steal the file. Change the mix. Keep your own picture. I am not posting a trophy chart from a machine you cannot see. # miss_bench.py # Lab harness. Treat printed rows as local output, not a benchmark paper. from __future__ import annotations import logging import time import tracemalloc from typing import Callable logging . basicConfig ( level = logging . DEBUG ) log = logging . getLogger ( " hot " ) HITS = { f " user: { i } " : i for i in range ( 800 )} KEYS = [ f " user: { i } " for i in range ( 1000 )] # 20% misses on purpose def lookup_except ( key : str ) -> int | None : try : return HITS [ key ] except KeyError : log . debug ( " cache miss key=%s " , key ) return None def lookup_get_quiet ( key : str ) -> int | None : return HITS . get ( key ) def lookup_get_log ( key : str ) -> int | None : value = HITS . get ( key ) if value is None : log . debug ( " cache miss key=%s " , key ) return value def run_mix (

AI Can Write Your Code. Can It Actually Debug It? AI coding assistants have changed how developers write software. You can describe a feature, generate a function, refactor a component, write a test, or explain an unfamiliar codebase in seconds. But there is one part of software development that is still surprisingly difficult: figuring out why something broke. Writing code and investigating a failure are two very different problems. When an application crashes, the answer usually isn't sitting inside the error message. You have to reconstruct what happened. The Problem With "Just Read the Stack Trace" Consider this Node.js error: TypeError: Cannot read properties of undefined (reading 'email') at getUser (/app/services/user.js:42:18) at processRequest (/app/controllers/auth.js:87:12) at async handler (/app/routes/auth.js:31:5) The immediate problem appears obvious. Something is undefined. But what caused it? Maybe: A database query returned no user. An API returned an unexpected response. Authentication middleware failed. A promise returned an unexpected value. A user record exists but its profile doesn't. An earlier function silently produced invalid state. The stack trace tells you where the program finally failed . It doesn't necessarily tell you where the bug began . That's the difference between error reporting and debugging investigation. AI Coding vs AI Debugging Most AI coding workflows look something like this: Developer ↓ Prompt ↓ AI ↓ Code Debugging is different: Failure ↓ Error ↓ Stack trace ↓ Execution path ↓ Application state ↓ Root cause ↓ Fix The AI needs to reason across that chain. Simply asking: "What does this error mean?" usually produces a list of possible explanations. That's useful, but it's not necessarily an investigation. A better question is: "Given this failure and its context, what is the most likely root cause, what evidence supports it, and how can I reproduce it?" That's a much more interesting problem for AI. A Simple JavaScript De

Have you ever watched an LLM endpoint return a clean answer that belonged to a different prompt entirely? I spent forty-eight hours blaming sampling noise, temperature, and a free model that would not sit still. The request logs looked honest enough, and the health check on the box stayed green the whole time. The bug was quieter than that: a cache key that hashed the user message and ignored everything else that actually changes a completion. I was trying to keep a small eval loop cheap, which is a very ordinary instinct. Free-model access is useful when you want overnight volume without treating every call as precious. I parked a thin HTTP wrapper on a free server, hashed each prompt, and stored the JSON body on disk so retries would not hammer the model. Does that sound reasonable? It did, until two different system prompts started colliding on the same key and I spent a day chasing "nondeterminism" that was just a hash. I ran that wrapper against MonkeyCode's free model access on the free server option because I wanted a boring place to reproduce the cache bug, not a production SLA. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing below depends on a named model, a quota, or a hardware claim. The lesson is the key function, and it still applies if you delete the product name from the stack. What I walked into The wrapper looked like every weekend cache I have written under time pressure. Incoming POST bodies were reduced to user_message , run through hashlib.sha256 , and written under ./cache/<hex>.json . A hit returned the file. A miss called the model, then wrote the file. I even logged X-Cache: HIT so future-me would feel scientific. That design has one attractive property and one fatal one. The attractive property is that identical user text becomes free after the first call. The fatal property is that user text is not the request. System prompt, temperature, stop sequences, tool schemas, and even a date injected into th

Article Summary: This article argues that EDR evasion testing does not depend on access to a cloud management console. Instead, the core requirement is a reproducible test environment employing single-variable controlled experiments to pinpoint detection triggers. It introduces the open‑source Detonator framework for automated EDR testing, supporting CrowdStrike and other major EDRs. The methodology emphasises a snapshot‑per‑run and one‑variable‑per‑test approach, and the article also notes the availability of authorised EDR test tokens. Categories: Cybersecurity, Red Teaming, Security Tools Executive Summary This briefing addresses a common challenge faced by red team practitioners: testing a new loader against a CrowdStrike‑protected endpoint without access to the cloud‑based Falcon console. Many assume that the absence of logs renders testing futile. However, the author contends that the console provides only a conclusion (whether the sample was killed), whereas what the tester truly needs is the point of failure in the attack chain. By adopting a reproducible, single‑variable testing methodology—supported by the Detonator framework—one can systematically isolate the specific API call or technique that triggers detection, independent of console logs. This approach transforms debugging from a blind guessing game into a rigorous scientific process. Introduction Consider the following scenario: you have obtained a CrowdStrike endpoint with the sensor online and fully operational. You are ready to test a loader that you have been refining for three weeks. Then you realise— you have no cloud console . No logs, no detection strings, no rule triggers. The immediate reaction is often panic: "How can I test without visibility?" But the central thesis of this article is that the console is not as indispensable as it seems. The true value lies not in receiving a binary verdict, but in understanding where and why the detection occurred. The Console: A Mere Reporter of Result

If you have ever called an OpenAI-compatible API with streaming enabled and received... nothing , you are not alone. No error, no exception — just a stream that "completes" successfully while your UI stays empty. After debugging dozens of these cases — and building an open-source toolkit to automate the diagnosis — I keep seeing the same four failure modes . Here is the field guide I wish I had. 1. Reasoning-only responses Some models emit their entire answer inside a reasoning channel (the "thinking" part) and mark the actual content channel as empty. The stream works . Token usage is reported. Your parser is happy. Your UI shows nothing. python # What arrives: {"delta": {"reasoning_content": "Let me analyze this..."}, ...} {"delta": {"content": ""}, "finish_reason": "stop"} The fix: always inspect the delta fields your model actually uses, not just content. If your client only reads choices[0].delta.content, a reasoning-only response is indistinguishable from an empty one. 2. Missing finish_reason When a proxy or router truncates the final chunk, finish_reason quietly disappears — and many client libraries silently drop the message instead of raising. The fix: treat a missing finish_reason as a red flag, not a quirk. Log it. Alert on it. A stream that ends without stop, length, or tool_calls did not end — it was cut. 3. Malformed SSE framing SSE looks trivial: lines of data: {...} ending with data: [DONE]. But: multi-byte UTF-8 characters can be split across chunk boundaries some proxies rewrite or strip the data: prefix chunks can arrive after [DONE], or the stream can end without it Each of these breaks parsers quietly — you lose characters in the middle of words, or the client hangs waiting for a terminator that never comes. The fix: log the raw frames before parsing. When something looks wrong downstream, the raw log is the only witness that tells the truth. 4. Truncated tool calls Agents assemble tool calls from multiple deltas. If the stream dies halfway, yo

A flat log can tell you that five things happened. It often cannot tell you which operation caused...

What actually changed You built a custom n8n node with its own credential type. The node works in a workflow. You open the credential in the n8n UI, click Test , and instead of a green checkmark you get: No testing function found for this credential. You double-check the node — credentialTest is defined in methods , testedBy is set on the credential declaration, everything compiles. n8n just refuses to see it. This is one of the most-reported custom-node issues in n8n's history — the original report is Stack Overflow q/75109822 and the underlying bug is tracked in n8n-io/n8n#8188 , with users still reproducing it on 1.58+ and 1.94 well after the original fix landed. The fix The root cause is not a missing function. It is in LoadNodesAndCredentials.ts : n8n generates nodesToTestWith in dist/known/credentials.json but only reads supportedNodes when linking a credential to its test function. For custom and community nodes the two keys never match, so the linkage is dropped and the UI shows the "no testing function" message. The fix that survives across n8n versions is to stop relying on credentialTest on the node and instead define the test directly on the credential class as an ICredentialTestRequest . Before — the linkage that breaks // credentials/MyApi.credentials.ts import { ICredentialType , INodeProperties } from ' n8n-workflow ' ; export class MyApi implements ICredentialType { name = ' myApi ' ; displayName = ' My API ' ; // ❌ testedBy points at the node's credentialTest, which the loader // never resolves for custom/community nodes. testedBy = ' MyApiNode ' ; properties : INodeProperties [] = [ { displayName : ' API Key ' , name : ' apiKey ' , type : ' string ' , typeOptions : { password : true }, default : '' , }, ]; } // nodes/MyApi.node.ts export class MyApiNode implements INodeType { methods : INodeTypeMethods = { credentialTest : async ( credentials ) => { // n8n never calls this for a custom node. const res = await fetch ( ' https://api.example.com/me '

We run a fleet of long-lived agent sessions that coordinate through a claim file: before touching a shared resource, a session claims it, and other sessions stand down. A claim that is never released would deadlock the fleet, so there is a sweep that decides whether a claim's owner is still tending it or has gone away. The sweep's core is a case over an exit code: mesh-mind-state " $win " > /dev/null 2>&1 case " $? " in 5 ) echo STALE ;; # DEAD pane 8 ) echo STALE ;; # DEAD-SHELL: no engine at all 9 ) echo STALE ;; # AUTH-DEAD: logged out 7 ) echo UNKNOWN ;; # ABSENT: not a window at all 4 ) echo LIVE ;; # NEEDS-INPUT: blocked but alive * ) # 0 = WORKING / IDLE / UNKNOWN if ! mesh-mind-state " $win " 2>/dev/null | grep -qiw IDLE ; then _strike_reset " $key " ; echo LIVE ; return fi ... esac Read the *) branch as what it actually is. It does not mean "the state is 0." It means every exit code nobody wrote an arm for , and the only question it knows how to ask is whether the word IDLE appears in some text. Anything that is not the string IDLE is treated as working . That is a classifier whose unhandled input votes healthy. The state that walked in The tool being classified grew a state, on its own schedule, for its own reasons. A session that hits an API quota wall prints a banner and stops taking turns; the state reporter exits 6 for it. # Exit (single window): 0 WORKING/IDLE/UNKNOWN · 4 NEEDS-INPUT · 5 DEAD · 6 RATE-LIMITED # 7 ABSENT · 8 DEAD-SHELL · 9 AUTH-DEAD There was no 6) arm. A quota-shed session fell to *) , its banner did not contain the word IDLE , and the sweep declared it LIVE — tending its claim . Nothing crashed. No log line said anything was wrong. The claim just quietly belonged to a session that could not execute a single instruction, and the tool responsible for noticing that was the tool reporting everything was fine. The bill: one claim sat there reading as merely expired for over five hours , while a nagging reflex kept sending its owner remind