标签：#debugging

Originally published on hexisteme notes . A predecessor note diagnosed three production features that passed every dedicated unit test and never executed at all, and why a unit test structurally can't see that gap. That note answered three cases I already knew about, because I'd already tripped over them. It didn't answer the question that matters once you've found three: how do you find the rest — the ones nobody happened to notice yet? This is that search: the tool that actually works, what it found across seven projects, and a fourth failure shape that the predecessor note's two fixes don't reach at all, because in that fourth shape the code was never the thing that was broken. The query, before the argument Before any of the specifics, here is the shape of the query, so you can run something like it against your own tables in under a minute: SELECT COUNT ( * ) AS total , SUM ( some_column IS NOT NULL ) AS filled FROM some_table ; If that comes back near 100%, this note may simply not apply to your codebase, and that's a real result, not a failure to reproduce it. Keep that in mind through the rest of this — every finding below is downstream of a query shaped like this one, not downstream of reading code and guessing. Grep is not the detector My first instinct, the same one the predecessor note's fixes point toward, was to grep for the failure shape — a default value, an unpopulated argument, a call site missing a keyword. In one afternoon it produced both a false positive and a false negative. The sharper miss: a literal grep for a write path failed to find an INSERT OR REPLACE statement that was, in fact, live and doing exactly the writing I was looking for. Grep matched the shape of the bug I expected walking in, not the shape the code actually had. Everything that survived scrutiny below came from asking a database a question, not from asking a shell how a string was spelled. The question that works is: of all the rows that exist, how many have this column fi

I have a small script, reply_comments.py , that pulls unanswered comments off my DEV.to articles and drafts replies to a markdown file so I can paste them in by hand. The API doesn't let a normal account post comments (that's its own bug I've written about before), so this draft-then-paste loop is the whole workflow. Every reply I've ever sent has come from reading the body field this script prints. Today I went looking for a bug distinct from everything already logged for this repo, and I ended up re-reading strip_html() , the function that turns a comment's raw body_html into the plain text I actually read: def strip_html ( h ): return re . sub ( r " \s+ " , " " , re . sub ( r " <[^>]+> " , " " , h )). strip () It does exactly one thing: strip HTML tags with a regex, then collapse whitespace. It's been in the file since the script was written and nobody had audited it on its own — every prior pass through this pipeline was about pagination, thread-depth walking, or dedup keys, never the text-extraction step itself. Here's the problem. DEV.to's API returns body_html as rendered HTML. A correct renderer has to HTML-entity-escape a commenter's own literal < , > , & , and quote characters, or they'd get mistaken for markup. So a comment that reads, in plain English: isn't it faster with a Q&A cache? Try List instead. comes back from the API as something like: <p> isn &#39; t it faster with a Q &amp; A cache? Try List &lt; String &gt; instead. </p> strip_html() 's regex only ever targets <[^>]+> — actual tags. It has no idea what to do with &#39; , &amp; , &lt; , &gt; . Those aren't tags, so the regex leaves them untouched. The whitespace collapse doesn't touch them either. What comes out the other end, into the exact field I read to draft a reply, is: isn&#39;t it faster with a Q&amp;A cache? Try List&lt;String&gt; instead. That's not a cosmetic nit. On a dev-focused comment section, & , < , and > show up constantly — generics, comparisons, "foo & bar," code snippets

Originally published on hexisteme notes . A status column in one of my agent fleet's ledgers said 30 items were queued to publish. A working session that day stated a backlog close to a month at the fleet's normal rate and deferred the work that keeps posts flowing into the queue. At that moment the ledger showed the same backlog. That exact numeric match suggests — but does not prove — that the ledger informed the decision. The real number of items actually waiting was 6. At one post published per day, that is six days of runway, against a low-water alarm configured to fire at 3. The gap came from a status value that was never advanced after publication, not from the queue-file count itself. A column just quietly stopped meaning what everyone assumed it meant, and by the time it mattered, it had been wrong for a while. The pipeline, briefly The fleet runs a small publishing pipeline: a draft gets written, a promotion step validates it and drops a file into a queue directory, and a scheduled job runs once a day, picks the oldest file in that directory, publishes it, moves the file into a published folder, and appends one line to a log. Alongside the queue directory sits a separate ledger: a flat TSV file, one row per item, with a status column meant to track where each item sits in its life — staged, queued, published. Two different things track the same concept: the files actually sitting in the queue directory, and a column in a table that is supposed to describe them. Where it broke Exactly one piece of code writes status=queued : the promotion step, at the moment an item enters the queue. Nothing else ever changes that value afterward. The daily publish job moves the file and writes to the log; it never opens the ledger. Nobody had assigned any code the job of setting the status forward to published . So queued stopped meaning "currently waiting." It came to mean "was queued at some point," which, once true, is true forever. Every item that had ever passed throu

You're debugging a broken SSO login. The identity provider (IdP) redirects back to your app, and somewhere in the request is a big blob called SAMLResponse . You grab it, Base64-decode it, and expect to see clean XML. Sometimes you do. Sometimes you get binary garbage that starts with bytes like 0x78 0x9c and looks nothing like markup. Both outcomes are correct. The difference is which SAML binding the IdP used, and once you know the two encoding chains, SAML debugging stops being guesswork. The two bindings, and their two encodings SAML sends its messages ( SAMLResponse , SAMLRequest ) using one of two HTTP bindings, and they encode the payload differently: HTTP-POST binding — the message rides in a hidden form field that auto-submits via POST. The value is simply: Base64(XML) Decode the Base64 and you get the assertion XML directly. This is the common case for the response coming back from the IdP. HTTP-Redirect binding — the message rides in a URL query string, so it has to be small and URL-safe. The value is: URLEncode( Base64( DEFLATE( XML ) ) ) That's three layers. If you only Base64-decode it, you're staring at the raw output of a DEFLATE compressor — which is exactly the binary garbage people report. This binding is typically used for SAMLRequest (the AuthnRequest your app sends to the IdP) and for Single Logout. Critically, the redirect binding uses raw DEFLATE (RFC 1951) with no zlib header and no checksum . That's the single most common thing people get wrong — they reach for a normal zlib/gzip inflate, it chokes on the missing header, and they conclude the blob is corrupt. It isn't; it just needs a raw inflate. Decoding both in Python import base64 import zlib from urllib.parse import unquote # --- HTTP-POST binding: Base64(XML) --- def decode_post ( saml_response : str ) -> str : return base64 . b64decode ( saml_response ). decode ( " utf-8 " ) # --- HTTP-Redirect binding: URLEncode(Base64(DEFLATE(XML))) --- def decode_redirect ( saml_param : str ) -> s

An LLM bill that doubles overnight has one of about eight causes, and the fastest route to it is not reading code. It is six queries over your request log, run in order, each of which eliminates a branch. The first one takes thirty seconds and settles whether you are looking for more requests or dearer ones. Before the queries: stop the bleeding If spend is still climbing while you investigate, put a ceiling on it first. A provider-side spending limit, a lowered rate limit on your own gateway, or disabling the newest feature flag all buy you time, and none of them require knowing the cause. Diagnosis is cheaper when the meter is not running. Resist the urge to change several things at once to make it stop. If you disable three suspects simultaneously and the spend falls, you have solved the incident and learned nothing, and it will return. What you need logged The runbook assumes one row per request. If you do not have this, building it is the first fix, and it is a day of work that pays for itself the first time this happens. CREATE TABLE llm_requests ( ts timestamptz NOT NULL , request_id text , model text NOT NULL , -- from the RESPONSE, the resolved one route text , -- which feature or endpoint caller text , -- service, job, or user id tenant text , -- customer, if multi-tenant prompt_tokens int NOT NULL , cached_tokens int , -- prompt tokens served from cache completion_tokens int NOT NULL , reasoning_tokens int , cost_usd numeric ( 12 , 6 ), -- computed at write time status int , attempt int , -- 1 for the first try, 2+ for retries duration_ms int ); Two columns do disproportionate work. attempt is what makes a retry storm visible instead of looking like organic traffic. And model taken from the response rather than the request is what makes an alias move visible — the request said one thing and the provider served another. The six queries, in order Volume or unit cost? Everything downstream depends on this answer, and it is one query. SELECT date_trunc('day',

I spent an evening opening other companies' product configurators — 3D and parametric tools on manufacturers' sites — looking for things that were genuinely broken. Twenty-seven of them. The findings were real. But the part worth writing down is that four separate times in one evening, my tooling told me an application was broken when it was fine. Every one of those four passed automated checks that looked rigorous. What caught them was a screenshot. If you write scripts that judge pages you don't own — uptime checks, competitor teardowns, scraping health, QA of an embedded widget — you will hit these. Here is the full list of signals that lied to me, and the one control that never has. The four false positives All four produced the same symptom: no <canvas> on the page, and an almost empty innerText . That looks damning when the page is literally titled "Configurator". It is also what three completely healthy situations look like: The tool starts on a click. An orange button launches it. My script measured an unopened door and reported an empty room. Four automated passes — raw HTTP with a browser UA, my own browser, two runs from a clean profile, a control on the same domain — all four confidently examined a page that hadn't started yet. The entire UI lives inside the canvas. One hall configurator draws its menus, its undo/redo and its PDF export in WebGL. Empty DOM text is correct there, not a defect. The tool is behind a login. I was measuring a sign-in page. Fifty-four characters of text and one button reading "Anmelden". The page is a landing page about the configurator, not the configurator. No network-level or DOM-level check distinguishes these from an actual failure. A screenshot distinguishes all four instantly. So the first rule I now follow, before any measurement at all: Take the screenshot first. Look at the picture. What you cannot see in the image, you do not measure. It costs one second and it is the highest-yield step in the whole process. The cor

While contributing to Grafana, I picked up a memory issue in the Time Comparison feature — a follow-up to earlier performance work I had done in the same area. A comparison panel was consuming significantly more memory than expected. The interesting part: the extra memory wasn't coming from real data. This post covers how I traced it to the root cause and fixed it. Background Time Comparison overlays an earlier period onto the current one — for example, this week vs. last week. The comparison data is fetched from the earlier window and shifted forward before rendering: Query → DataFrame → Prepare frame → Shift → Render │ └─ Gap filling The important detail: gap filling ran before the comparison frame was shifted. The Problem I reproduced the issue with: Parameter Value Series 500 Window 6h Interval 20s Compare offset 24h A single-period panel contained roughly 540,000 points , so a comparison panel should be about 2× the baseline . Instead, the compare frame contained 3,240,500 points — ~6× the baseline — and consumed 76.4 MB . The question was: where did the extra points come from? Investigation I first verified the baseline to rule out the query returning unexpected data. It was correct. Then I used a reproducible browser harness and a heap snapshot to inspect the extra memory. Most of it was null rows introduced during gap filling — not real samples, not copies. Following the frame through the preparation pipeline revealed why. When gap filling ran, the compare frame still represented data 24 hours in the past , but the gap-filler was using the current time range as its reference: Compare frame Current range [===== 6h =====] [===== 6h =====] └─────────────── 24h ───────────────┘ gap-filler reads this offset as one gap At a 20-second interval, 24 hours is: 24 × 60 × 60 / 20 = 4,320 intervals So up to 4,320 null positions per series were introduced purely because the frame hadn't been shifted yet. The frame was then shifted forward, leaving most of that padding out

The most consequential bug in this project had been there since the beginning, survived several full end-to-end runs, and was reported as a PASS every time. ✓ PASS slow-and-low detected within 30m (7.3m), never exceeding legit rate That line is true. The scorer flagged it correctly, well inside the bound. What the line doesn't say is that the attacker was served every single request it ever made . Zero non-allow decisions, across the entire scenario. Detected and never once stopped. This is one instance of a pattern that accounts for more real bugs in this project than every other cause combined: a component contributes nothing, no error is raised, and every surrounding number stays plausible. The bug The scorer computes windows at 1m, 5m and 1h, and publishes each result to a per-client key in Redis and OPA. Each result. To the same key. So the last writer won. And 1-minute windows close most often, so they always won. slow-and-low issues about two requests a minute. Its 1m windows fall below the minimum request count and score zero. Its 5m and 1h windows accumulate the miss ratio that earns a deny . Every one of those zeroes immediately overwrote the deny. The entire premise of a multi-scale pipeline — that different attacks are visible at different scales — was silently violated by the publication step. Any detection that only appeared at a coarser scale was discarded. The fix is a roll-up: publish the most severe verdict across window sizes within the freshness horizon the policy already uses. Afterwards, the same attacker is denied on 24–31 of its 44 requests. Why it survived so long Because the report could not express it. Detection latency was computed as the earlier of two very different facts: the scorer's first non-allow window, and the gateway's first non-allow decision. Printed under one heading — detected — a client that was noticed but never touched looked identical to one that was noticed and blocked. A report that averages over the distinction you ar

Every anomaly detector answers one question: compared to what? That comparison, the baseline, is where I lost the most time on this project, and every failure had the same signature. Nothing errored. No test went red. The numbers stayed plausible. The detector just quietly stopped detecting. Four of them, in the order I found them. 1. The peer group contained the client it was judging Cold-start clients have no history, so they're compared against a pool of other clients' recent benign windows. Reasonable. The pool was keyed by feature: private readonly peer = new Map < FeatureKey , number [] > (); Every benign window every client produced went into the pool that client was later compared against. Including itself. So a client could define its own normality . Feed in enough windows and any behaviour becomes unremarkable — which is precisely the cold-start attacker the layer exists to catch. What made me look was not reasoning, it was an experiment that wouldn't sit still. I was trying to build a demo client that reliably landed in the middle of the response ladder, and holding the traffic shape fixed while changing only the request interval flipped the outcome between allow and step_up : gap=500ms origins=5 → allow (peak 0) gap=700ms origins=5 → step_up (peak 83) gap=800ms origins=5 → allow (peak 0) A knife edge like that is never a tuning problem. The outcome depended on a race between a client's own samples reaching the pool and the pool being consulted. Fix: key the pool per client, and exclude the client under evaluation. for ( const [ clientId , values ] of byClient ) { if ( clientId === excludeClientId ) continue ; // this is what "peer" means … } Afterwards the behaviour became monotone in the actual evidence, and identical at every request interval: origins 1 3 4 5 6 peak score 17 35 59 83 100 tier allow log throttle step_up deny Lesson: if a parameter that shouldn't matter changes the outcome, stop tuning and go find the defect. Knife edges are symptoms. 2.

Point a model at your repositories, ask why something broke, and you get an answer. Coherent, names a mechanism, shows you how it got there. The working is the problem rather than the reassurance. It looks the same whether it ends at a value in a record or at what usually happens, and only one of those is evidence. I ran the same question at four levels of evidence, on two systems, across three models. The question was never find me a bug. It was always this already happened, work out how. The pass that worries me is the one just short of enough evidence. There is enough there to build a real hypothesis, and not enough to tell a real one from a plausible one. Two of its guesses sent me to look at things I had not checked. One was a dead end. The other was a real problem I had not known about, and nothing in either answer told me which was which. A good guess costs the same to chase as a real one. You find out which it was at the end. The four rungs, and what each one removes I stopped thinking about this as adding context. Each rung takes away something the model would otherwise have to guess at, and that is the more useful way to look at it. One. Repository access, broad. Point it at everything and ask. It removes nothing. The answer was structurally sensible, named components that were not involved, and arrived with no more hesitation than the correct one did three rungs later. If your team tried this once, got something confident and wrong, and decided the tooling is not there yet, this is probably where you stopped. Two. Three bounded repositories, plus a written map. Which service talks to which, over what protocol, with what delivery and ordering guarantees. This removes rediscovery. What made it work was not the map. It was telling the model to treat the map as true and not go and check. That bought focus and gave up verification. Maps go stale quietly, and I have removed the step where it might have noticed. Three. Plus traces and logs covering real executio

Hard debugging is rarely defeated by a lack of tools. It is defeated by three quiet habits: assuming the fault is where the symptom appears, clinging to the first explanation, and hoping a tool will do the thinking. A difficult fault is usually lost to those habits before you read a line of code. The engineers who resolve hard faults are the ones who notice these defaults and replace them with a patient, evidence-first mindset. Most hard bugs are lost before we touch them, in the attitude we bring to the session. When something breaks, the average person rushes in with three quiet habits: they assume the fault lives exactly where it shows up, they cling to the first explanation their mind offers, and they hope a tool or a smarter person will tell them what to do next. Those habits feel natural, but on hard faults they are exactly what keep us stuck. Put two engineers on the same failing board. One finds a way through in an afternoon; the other is still going three days later. The difference is rarely raw intelligence or how many commands they know. It is the mental posture each brings to the work before the first step. Handling a hard debug session is less about knowing every tool and more about managing your own assumptions, reactions, and impatience. A tough problem is usually lost in your mindset before it is lost in your methods. Habit one: starting too narrow The first habit is to fix on the most visible symptom and refuse to look anywhere else. Something breaks, so we stare at the last thing we changed, and we return to it because it is familiar and close at hand. When the answer is not there, we look harder in the same place instead of stepping back. Here is what that looks like on real hardware. A device keeps dropping off the bus. You are a kernel person, so you open the driver and read it, carefully, for three days: the probe path, the error handling, the power-management callbacks. Every line is correct, and the device still fails. The fault was a layer b

The owner had already asked for the alert emails to stop. A fix shipped. Then another email landed. Then another. "ong it just ssent me abother email," he said, voice-dictated, unedited. Fifteen minutes later: "go another one." The system was reporting an outage that did not exist. The Transport That Only Ever Failed A 14-PR merge train had just moved every cron producer's alerting off shared email and onto Buzz, a Nostr-relay team chat. One producer per PR, each with its own liveness contract and a bead receipt. It shipped cleanly. But the library backing those producers carried a default that had only one job: fail. AF_BUZZ_CMD = " ${ AF_BUZZ_CMD :- af_default_buzz_post } " af_default_buzz_post returned 1 with "no Buzz transport injected". Every caller that sourced the library (which is every cron producer) exhausted its Buzz retries and fell through to the email floor. The system reported a false Buzz outage while the relay was healthy. It did this 2 to 5 times per hour. Evidence arrived in the logs: 581 dedup markers, a steady stream of "[INTENT ALERT FLOOR: Buzz unreachable]" emails, and sweep.log showing buzz=ok only for the handful of callers invoked through the CLI entrypoint rather than by sourcing the library. That asymmetry was the bug. The CLI had a one-line fixup swapping in the real transport, annotated in a comment as "the library path is unchanged". The library path did not, and the cron producers all take the library path. The fix promoted the real transport to the default for both seams. af_buzz_transport already discovers the installed buzz-notify.sh and already fails closed when it is genuinely missing. The dead CLI fixup was deleted. Fail-closed behavior survives, but now it is conditional on genuine absence rather than on every caller remembering to opt in. Why not migrate callers one at a time? Because the per-caller route leaves the next new producer to rediscover this the same way. Flipping the default fixes the class, not the instance. The

Start with the Built-in Inspector Before reaching for external tools, remember Node.js has a built-in debugger. Run your script with --inspect and open chrome://inspect in Chrome to get a full DevTools experience: breakpoints, step-through, console, and even memory profiling. node --inspect app.js For a quick breakpoint without touching the browser, use --inspect-brk to pause on the first line. This is great for debugging startup issues. Use debugger Statements and Conditional Breakpoints Sometimes you need a breakpoint only when a condition is true. Instead of littering your code with if blocks, set a conditional breakpoint in DevTools. Right-click the line number, choose "Add conditional breakpoint," and enter an expression like user.id === 42 . For quick inline debugging, debugger; works but remember to remove it before committing. I often use it temporarily when I'm too lazy to open the DevTools UI. Log Like a Pro with util.inspect console.log of an object prints [object Object] which is useless. Use util.inspect with depth and colors to see nested structures clearly. const util = require ( ' util ' ); console . log ( util . inspect ( myObject , { showHidden : false , depth : null , colors : true })); Or in modern Node, you can use console.dir with { depth: null } for the same effect. Async Stack Traces: Don't Lose the Context Async errors are painful because stack traces often end at the event loop. Node 12+ gives you better async stack traces by default, but you can improve them further by using Error.captureStackTrace in your own error classes. class MyError extends Error { constructor ( message ) { super ( message ); Error . captureStackTrace ( this , MyError ); } } This makes the stack trace point to the caller, not the constructor. Handle Unhandled Rejections and Exceptions Silent failures are the worst. Set up global handlers to log errors properly and exit gracefully. process . on ( ' unhandledRejection ' , ( reason , promise ) => { console . error ( ' U

Originally published on hexisteme notes . Every test passed. The page was in pieces. I was rebuilding a small internal dashboard — FastAPI, Jinja2 templates, hand-written CSS, no build step — and the layout had come apart. Timeline rows unstacked into a vertical column. Status dots floated free of their rows. Log group labels overlapped. It looked exactly like a page whose stylesheet had failed to load. The stylesheet had loaded. All 481 tests in the suite were green. And when I grepped the CSS file for the rules that were obviously not being applied, they were sitting right there on disk, correctly written. The cause was a comment closer. Somewhere in the middle of the file, a /* had been closed with #} — Jinja's comment terminator — instead of */ . Muscle memory, from switching back and forth between .html templates and .css . CSS then did precisely what the specification tells it to do: it kept reading. The comment ran on and swallowed the next 15 lines of rules — the timeline-row grid, the feed, the bucket layout, the dot alignment — until it hit the next real */ , seventeen lines down. No error. No console warning. No failing test. The rules were present in the file and absent from the page at the same time. The typo is the least interesting part. What's worth keeping is why CSS is designed to fail without symptoms, why source-level review cannot see it, and why the fix is a two-line assertion rather than more care. Why nothing complained: CSS has no fatal errors The CSS Syntax specification defines comment consumption like this: on seeing /* , consume everything "up to and including the first */ , or up to an EOF code point." No notion of a comment being too long, no heuristic about blank lines or braces, no upper bound. First */ wins. A comment closed seventeen lines later than intended is not a malformed comment — it is a well-formed comment that happens to be seventeen lines long. The parser has no way to know you meant something else. Even if the comment h

Originally published at wostal.eu . TL;DR : I handed a CI re-platform — GitHub Actions → Argo Workflows, GitHub → Codeberg — to an AI agent. The rewrite looked correct and passed lint, but it didn't account for one thing: the runtime environment had changed completely. The old runner was an external VM; the new one is a pod inside the cluster, where Tailscale IPs aren't routable. Nobody ran the pipeline end-to-end, so every difference stayed as a landmine — six of them. The worst part wasn't the AI's mistake. It was that I spent hours blaming the network when the real culprit was a single missing newline in an SSH key. This is the companion to a different incident. While I was migrating my homelab k3s control plane from SQLite to etcd , the pipeline that ran that migration turned out to be broken in ways that taught me more than the migration itself. In this post I'll cover: Where the mess came from — an AI-assisted CI re-platform that nobody validated end-to-end The phantom CNI bug — hours of chasing MTU, Tailscale, firewall, and conntrack The real culprit — a single missing newline in an SSH key Why early failures hide later ones — six latent bugs, surfaced one at a time The lesson about delegating migrations — to a human or an AI Where the mess came from The entire CI had just been migrated from GitHub Actions to Argo Workflows , and from GitHub to Codeberg — and that migration was handed to an AI agent 🙂. The re-platforming rewrote the logic roughly 1:1, but it didn't catch that the runtime environment had changed fundamentally . The old runner (GitHub Actions / ARC) behaved one way; the new one is a pod in the cluster — from which Tailscale IPs aren't routable, but private IPs are. Nobody (human or AI) ran the pipeline end-to-end, so all of those differences stayed as landmines. That's why there were six latent bugs and the hang I'm about to describe — not random faults, but traces of a GHA→Argo translation that was never validated in the new environment. The p

Two people asked a context compressor two completely different questions. It gave them the same answer. Not a similar answer — byte for byte the same 544 characters. Here's what that looked like: query="Fix the IntegrityError on commit" level=L0 -> 159 tok cache_hit=False query="Explain the tax rounding TODO in compute_tax" level=L3 -> 159 tok cache_hit=True identical output: yes (544 chars both) Different question. Different compression level. Same 544 characters, served from cache. Finding it I wasn't looking for this. I was auditing something else entirely — measuring how much meaning a context compressor loses, not how fast it runs. My harness feeds the same corpus through the compressor with different queries and checks which critical substrings survive: file paths, error types, line numbers, identifiers. I noticed two rows in my results table were identical. Same token count, same output. My first assumption was that my own harness had a bug — that I was passing the same query twice and hadn't noticed. So I changed the second query to something with no words in common with the first, and bumped the compression level from L0 to L3, which should change the output dramatically on its own. Same 544 characters. That was the moment it stopped being my bug. The cause One line: sid = content_hash(content) That sid was doing two jobs. It was the shadow ID — the handle used to refer to a stored document. And it was also the cache key. As a shadow ID it's correct: the same content should get the same handle. As a cache key it's wrong, because the output of compress() doesn't depend only on the content. It depends on the content and the query and the compression level. Two of those three inputs were simply not part of the key. So the first caller warmed the cache for a piece of content, and everyone who touched that same content afterwards got the first caller's answer — regardless of what they actually asked for. Why this is worse than a stale cache A stale cache gives y

I have a small script, reply_comments.py , that keeps me from having to re-scan every DEV.to article for new comments by hand. It has two commands: pending (unanswered comments I haven't drafted a reply to yet) and audit (drafted replies I said I'd paste manually but apparently never did). I've already fixed two bugs in this file — one in needs_reply() (a thread stayed "handled" forever after a single reply, even when the other person followed up again) and one in audit() (it only checked direct children, so a reply nested two levels deep was invisible). Today I found a third, in pending() itself, and it's the kind of bug that hides precisely because the first two fixes made everything else in the file look trustworthy. What pending() actually does Comments on DEV.to come back from the API as trees — each top-level comment has a children list, and replies can nest arbitrarily deep. pending() walks each article's top-level comments and decides, for each one, whether it needs a reply: def pending (): try : drafted_text = open ( DRAFTS , encoding = " utf-8 " ). read () except FileNotFoundError : drafted_text = "" drafted_codes = set ( re . findall ( r " ^## (\S+) " , drafted_text , re . M )) out = [] for a in api ( f " /articles?username= { ME } &per_page=100 " ): if not a [ " comments_count " ]: continue for c in api ( f " /comments?a_id= { a [ ' id ' ] } " ): if not needs_reply ( c ): continue if c [ " id_code " ] in drafted_codes : continue out . append ({ " id_code " : c [ " id_code " ], " author " : c [ " user " ][ " username " ], " article " : a [ " title " ], " comment_url " : f " https://dev.to/ { ME } /comment/ { c [ ' id_code ' ] } " , " body " : strip_html ( c [ " body_html " ]), }) return out needs_reply(c) is the fix from a few weeks ago — it recurses the whole subtree and checks who posted the most recent message, not just whether I've ever replied. That part's correct. The bug is in the two lines right after it: c["id_code"] and c["body_html"] . c here i

On Windows I kept launching background servers from a task runner and watching them die the instant the launching step finished — no error, no log, just gone. The task runner was wrapping everything in a job object, and job-object teardown kills every child process on return. Nothing I did inside the child mattered; its death warrant was signed by how it was born. The workaround was to have the process created by something that outlives the runner — the OS scheduler, a WMI process-create call — instead of spawning it as a doomed descendant. When a process keeps dying without a trace, look at its lineage before its code — some parents kill their children on the way out, and no amount of hardening inside the child fixes how it was spawned.

When I published v2.0.0 , I thought the hard part was over. The CLI could turn Claude artifacts into native Windows applications in a single command. It worked on my machine, the demos looked great, and I was excited to share it. Then people started using it. That's when I realized the difference between building software and shipping software . The bugs only users can find Most of the problems weren't visible in a five-minute demo. Some applications worked perfectly. Others silently lost all of their data after being closed. Some builds succeeded exactly once. The second build failed unless the runtime cache was manually deleted. Everything looked correct... until someone actually tried using it. The rabbit hole What I thought would be a quick patch became a deep dive into things I'd never touched before. Over the course of v3 I ended up learning far more than I expected: Git branching and release workflows Semantic versioning GitHub Actions npm packaging npm pack Runtime integrity verification SHA-256 checksums Neutralino runtime management JavaScript debugging Integration testing None of those were part of the original project idea. Persistent storage finally works This was the biggest challenge. Artifacts using localStorage behaved differently depending on where they were running. Inside Claude everything worked. Inside the generated desktop application... not always. Tracking this down took far longer than writing the original feature. The result is that applications now persist their data correctly across launches without requiring any code changes. If your artifact uses localStorage , it should simply work. Runtime improvements The runtime layer also received a major overhaul. Version 3 now handles: automatic runtime downloads integrity verification cache recovery consecutive builds without manual cleanup Those aren't exciting features to demo, but they're exactly the kinds of improvements that make a tool feel dependable. Better testing I also spent a lot mo

This is part of my Building Fluentic Style series, where I’m writing down the design decisions, tradeoffs, and small surprises from building Fluentic Style . It is one thing to make a styling library feel good in a client-side app. It is another thing to make it feel good in Next.js App Router. In a simple SPA-style development setup, most of the styling loop lives in one place: component renders in the browser Fluentic style chain resolves atomic CSS rule is inserted DevTools can inspect the generated rule sourcemap points back to authored code That is already a lot of work. But at least the browser is the main place where the style is produced and consumed. Next.js App Router changes the shape of the problem. Now the page can involve: server rendering React Server Components client components streamed HTML hydration client-side navigation HMR Webpack or Turbopack development sourcemaps production extraction So the hard part is not just “can Fluentic run in Next.js?” The hard part is: Can Fluentic keep the same CSS debugging experience when styles cross the server/client boundary? That is what this post is about. Docs for the Next.js integration are here: Next.js Integration DevTools And Sourcemaps Runtime And Dev Debug Without Getting Lost The Goal Was Not A Special Next.js API I did not want Fluentic to have one mental model for client apps and another one for Next.js. This should still be normal Fluentic: const card = style ({ padding : 16 , borderRadius : 12 , }). hover ({ boxShadow : ' 0 12px 30px rgb(15 23 42 / 0.16) ' , }); export function Card () { return < section css = { card } > Hello </ section >; } And this should still be normal Fluentic too: const buttonStyles = { root : style . slot ({ display : ' inline-flex ' , border : 0 , }), label : style . slot ({ fontWeight : 700 , }), }; const danger = style . scope ([ buttonStyles . root ({ backgroundColor : ' #dc2626 ' , }), buttonStyles . label ({ color : ' #ffffff ' , }), ]); The Next.js integration shou