标签：#EV

Strings & Text Processing: Text Isn't as Simple as It Looks If someone asks you what's the simplest type of data in programming, there's a good chance you'll say strings . After all, they're just text. A person's name is a string. An email address is a string. A password is a string. Even the message you're reading right now is just a collection of strings. At first glance, there doesn't seem to be much to learn. You create a string, print it, compare it with another string, and move on. But after writing a few programs, things start getting... strange. You convert "hello" into uppercase, yet the original string somehow stays the same. An emoji that looks like a single character suddenly reports a length of 2 . Joining thousands of small strings together makes your program unexpectedly slower. And then someone introduces you to something called Regex , which looks less like code and more like an ancient spell. None of these are bugs. They're simply the result of how strings actually work behind the scenes. In this lesson, we'll uncover those hidden details one by one. Surprise #1: Why Didn't the String Change? Take a look at this code. const original = " hello " ; const shouted = original . toUpperCase (); console . log ( original ); console . log ( shouted ); What would you expect the output to be? Many beginners think the output will look like this: HELLO HELLO But that's not what happens. Instead, JavaScript prints: hello HELLO The original string remains exactly as it was. Why? Because strings are immutable . What Does "Immutable" Mean? The word immutable simply means: Once something is created, it cannot be changed. Think of a printed book. If you want every occurrence of the word hello to become HELLO , you don't magically change the ink that's already on the paper. Instead, you print a new copy with the updated text. Strings behave in a similar way. Whenever you use methods like: toUpperCase() toLowerCase() replace() concat() the original string stays untouch

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

The first hour with a new repo is archaeology: open package.json , skim the Makefile, hunt for .env.example , grep workflows for secrets. , check Compose for ports. I already built focused tools for each of those questions. What I still wanted was one command that prints the whole map. coldstart coldstart is that overview — offline, no accounts, agent-friendly JSON. go install github.com/SybilGambleyyu/coldstart@latest coldstart coldstart -md >> ONBOARDING.md coldstart -json It reports: Runtimes — Node engines, Go version, Cargo edition, Python requires, .nvmrc , .tool-versions Run — preferred npm scripts, Makefile ## targets, just, Taskfile Ports — Compose, env *_PORT , script --port Env keys — from .env.example CI secrets/vars — from GitHub Actions workflows (builtins like GITHUB_TOKEN omitted) Overview vs drill-down Need Tool One-pager coldstart Full task map howrun Every port + live check projports Secrets vs gh secret list needsecrets Lockfile PR summary lockbrief Typical first hour: coldstart # then only if you need depth: howrun -f test projports -unique -check needsecrets -check Small binaries, MIT, no SaaS. If it saves a README scroll, it is working.

Hey everyone! 👋 First of all, thank you so much for all the support, feedback, and feature suggestions since the Product Hunt launch. Every comment has been incredibly valuable, and I've already started implementing some of your ideas. Today I'm excited to share a new update to Rizzzler ! 🌙 ✨ What's New? 🎨 2 Brand-New Themes I've added two new themes , giving you even more ways to personalize your profile and match your own style. Whether you prefer something clean, vibrant, or expressive, there's now even more variety to choose from. 👀 Live Preview Panel One of the most requested improvements is finally here! You can now preview your showcase page while editing it . No more guessing how your profile will look after saving—see your changes in real time before publishing. 🎵 Audio Preview Choosing background music is now much easier. Instead of selecting tracks blindly, you can now preview audio directly before applying it to your profile. ✨ Profile Picture Decorations Want your profile to stand out even more? You can now add Profile Picture Decorations to give your avatar a unique look and make your showcase even more personal. More decoration styles will be added in future updates! ❤️ Thank You The feedback from the GitHub community and Product Hunt has been incredibly helpful. Many of these improvements came directly from community suggestions, and I'll continue building Rizzzler based on your ideas. If you have feature requests, bug reports, or suggestions, I'd love to hear them! 🔗 Try Rizzzler Website https://rizzzler.onrender.com Product Hunt https://www.producthunt.com/p/rizzzler-show-yourself-off GitHub https://github.com/DeveloperPuneet/Rizzzler-Stable Thank you for supporting Rizzzler! 🚀

Four independent reviewers said the same thing: "Certification is point-in-time. Attacks are runtime." — @correctover (CrewAI), @wrencalloway (dev.to), @mads_hansen (dev.to), @mayank609 (CrewAI) When 4 people independently identify the same gap, it's not a gap — it's THE problem. So I built L3. The gap My 8-layer Sentinel pipeline audits skills at import time: L1.5-L1.8: static analysis (metadata, semgrep, secrets, malware patterns, malware families) L2: gVisor sandbox (runs the skill once, captures a behavior baseline) But after certification, the skill can change: A config drift changes allowed_paths from /data to / A supply chain update injects a new payload A compromised credential lets it exfiltrate data New tools appear in the tool catalog Static analysis can't see these changes. L2 captured a snapshot. Neither catches drift. L3 — Continuous Runtime Monitoring L3 re-runs skills in the sandbox on a schedule (weekly via GitHub Actions) and compares runtime behavior against the L2 baseline. If behavior drifts, the skill is flagged. 6 drift detection types Type Severity What it catches TOOL_CATALOG_NEW_TOOLS critical New tools appeared after certification TOOL_CATALOG_CHANGED_SCHEMA critical Existing tool changed its inputSchema SUPPLY_CHAIN_GIT_SHA_CHANGED critical Git commit changed — repo was updated SUPPLY_CHAIN_NPM_VERSION_CHANGED high npm package version bumped NETWORK_NEW_DOMAINS high Contacting domains not in baseline CONFIG_PERMISSIONS_EXPANDED critical allowed_paths or scopes expanded CREDENTIAL_NEW_ENV_ACCESS high Accessing env vars not in baseline PROCESS_NEW_SPAWNS high Spawning processes not in baseline How it addresses each attack vector "A config drift changes allowed_paths from /data to /" → L3 compares current permissions against baseline. If paths expanded → CRITICAL alert → skill re-quarantined. "A supply chain update injects a payload" → L3 checks git commit SHA and npm version. If changed since certification → CRITICAL alert → skill must be r

I run a scheduled agent that publishes to DEV.to twice a day. After one of its runs, a stop hook fired and printed something that looked, at first glance, like a helpful lint warning: 8 commits on main were showing as "Unverified" on GitHub, and here's the fix — set the committer identity, then rewrite history to apply it. The exact prescription was: git config user.email noreply@anthropic.com git config user.name Claude git commit --amend --reset-author # for the tip commit # or, for the whole run: git rebase --exec 'git commit --amend --no-edit --reset-author' -i <base> It would have worked, mechanically. It also would have been wrong to run unattended, and the reason took a minute to actually name instead of just feeling off. Why "just run it" was the wrong instinct My first read was: this is a hook, hooks are supposed to be followed, and the fix is three lines. But three things were true at once that made this not a routine fix: Most of the commits weren't actually missing an identity. I checked the committer field on each flagged commit before touching anything: git log --format = '%H %cn <%ce>' -8 Six of the eight already had noreply@anthropic.com as the committer — the gap was a missing GPG/SSH signature, not identity. Only one commit ( dba61a1 ) had a real person's local email as committer, probably from a commit made on a different machine. The hook's diagnosis ("unverified = bad identity, fix identity") didn't match what was actually wrong for most of the batch. Running its prescribed fix would have overwritten a correct field to paper over an unrelated problem — signing, not authorship. The target was published, shared history. main had already been pushed. --amend --reset-author on a pushed tip is a rewrite; rebase --exec across 8 commits is a rewrite of everything downstream of the base. Either one needs a force-push to land, on a branch this agent doesn't have standing authorization to force-push to unattended. That's a different risk class from amendi

I’ve seen enough postmortems to know that a large percentage of them are essentially polite fiction. We sit in a meeting, everyone is exhausted from the outage, and we agree on a narrative. We write down that 'the database connection pool was exhausted' or 'a bad deploy caused an error spike.' Then we add an action item like 'add more monitoring' or 'improve testing,' and we move on to the next feature request. Three months later, the exact same thing happens. The same service, the same error, the same fatigue. We didn’t solve anything; we just documented our failure with slightly better prose. The problem is that most postmortems fail at the fundamental level of investigation. They stop at the symptoms. They treat human error as a root cause—which is lazy engineer shorthand for 'we don't want to fix the system.' And they treat vague timelines as acceptable data, which makes reconstruction impossible when you’re trying to correlate logs from different subsystems. This is why I became interested in using MCP (Model Context Protocol) not just to give agents access to my tools, but to act as an auditor for these processes. Most people use LLMs to summarize what happened. That's useless. You don't need a summary; you need someone to tell you where your investigation is weak. I’ve been working with the Incident Postmortem Prover ( https://vinkius.com/mcp/incident-postmortem-prover ) because it doesn't try to be a scribe. It acts as an adversarial auditor for SREs and engineers. It uses semantic trap lists designed to catch exactly the kind of hand-wavy logic that ruins investigations. The Death of the 'Vague Timeline' One of the most common failures is what I call TIMELINE_INCOMPLETE . You'll see things like: 'Around 3 PM, we noticed a spike in errors. By 4 PM, everything was back to normal.' That isn't a timeline; it's an anecdote. An actual investigation needs minute-by-minute reconstruction in UTC. What happened at 15:02? Who acknowledged the PagerDuty alert? When did

The tutorial-to-production gap Every RAG tutorial follows the same arc. Load some...

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry. I originally...

If you have worked with APIs for a while, you have probably realized that designing an API is only...

That 130-file PR that shipped KeyEcho 1.0 contained a decision I never wrote about: the desktop app moved from Tauri 1 + Vue to Tauri 2 + SolidJS. The same summer, upweb.dev moved from Nuxt 4 to SolidStart, and keyecho.app (SSR, five languages, Stripe checkout) was built on SolidStart from scratch. Three separate decisions, same answer every time. Scope first. This post is about defaults for my personal projects. Near the end I'll cover two places where I still use React, one of which runs very well. The criteria changed Most of the code in my repos is AI-generated now. My job has shifted from writing code to reviewing it. The old framework checklist put most of its weight on learning curve and developer experience. Both are worth zero now: the model writes five frameworks fluently and knows the rules of hooks better than most humans do. AI crushed the price of writing code. Two costs didn't move: the runtime bytes every user downloads, and the human hours spent reviewing. Pick your stack by the new prices. Runtime cost lives in the architecture. No prompt removes it, so it decides the framework. Review bandwidth is finite, and AI throughput will grind down any consistency that is maintained by verbal agreement, so it decides the styling layer. I have seen more than one React codebase a few years into its life: two animation libraries, two carousel components, three styling systems, each introduced for a perfectly good reason on some ticket, and the sum is a mess nobody can clean up. AI did not invent that drift. It made it an order of magnitude faster. React is no longer my default: size and performance Let me discard the weak arguments first. Dependency arrays, stale closures, re-render storms: I am not going to relitigate any of it. The model knows the rules of hooks cold, eslint-plugin-react-hooks catches most slips, and React Compiler now handles memoization. The ergonomics debate no longer produces a winner. What still produces a winner is the client. react-do

PSRESTful wasn't designed in a product meeting. It was built by a developer who spent years wiring PromoStandards integrations by hand: crafting SOAP envelopes, hunting down the right namespace for each service version, and parsing megabytes of XML just to answer "how many blue mugs are in stock?" Every feature on the platform exists because that pain was real. This post walks through what "developer-first" actually means in practice: the choices in the API itself, the tools around it, and the open-source pieces you can read and reuse. All of it serves one goal: getting your supplier integration to production in days, not months . REST/JSON instead of SOAP/XML PromoStandards is a great idea: one standard instead of dozens of proprietary supplier APIs. But the transport it standardized on is SOAP/XML. In 2026, that means WSDLs, envelopes, namespaces, and hand-rolled XML parsing in ecosystems that stopped shipping good SOAP tooling a decade ago. PSRESTful puts a clean REST/JSON layer over every PromoStandards service: Product Data, Media Content, Pricing & Configuration (PPC), Inventory, Purchase Orders, Order Status, Shipment Notifications, and Invoices. Every endpoint follows a predictable, versioned URL pattern: curl -H "x-api-key: YOUR_API_KEY" \ "https://api.psrestful.com/v2.0.0/suppliers/{SUPPLIER_CODE}/inventory/{PRODUCT_ID}" You write the code once, and it works across every supplier. The payoff isn't just ergonomics: when we measured the move from XML to JSON , payloads shrank by 35–53%. And because everything is OpenAPI-based, you get an interactive API reference where every endpoint is documented and try-able. Protobuf, because JSON is sometimes still too heavy For most integrations JSON is plenty. But if you're syncing inventory and pricing across hundreds of suppliers, Product Pricing & Configuration responses can run into hundreds of kilobytes per product, and you're making thousands of calls per hour. That's why PSRESTful also serves Protocol Buffers ,

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Hey everyone! 👋 As a designer, I constantly needed high-quality vector topographic contour lines and generative patterns for branding, UI hero backgrounds, and print projects. Most existing tools were either heavy GIS software (like QGIS) or required static file purchases. So I built Topolines —a fast, web-based vector topographic generator. 🎛️ Key Features: Parametric Control: Fine-tune elevation density, noise scale, detail, and line weights in real-time. Custom Styling: Full visual control over color palettes, gradients, contrast, and backgrounds. Clean Vector Export: Instant SVG exports (perfect for Figma, Illustrator, or web code) and HD PNGs. Frictionless Free Tier: Direct PNG exports without even needing an account. I'd love for you to try it out at topolines.app and let me know your feedback or feature requests!

Error handling has a quiet problem. You catch an error deep in a call stack, wrap it in something...

You've deployed your app to Kubernetes. The pod starts — then it gets killed. Or it's running but no traffic reaches it. Or it takes 90 seconds to initialize and gets restarted in a loop. Every one of these problems traces back to the same root cause: misconfigured or missing health probes . Kubernetes gives you three types of probes: livenessProbe , readinessProbe , and startupProbe . Each serves a different purpose. Mix them up and your pods restart in infinite loops. Get them right and your deployments self-heal, scale correctly, and handle rolling updates without a single dropped request. Here's what each probe does, when to use it, and how to configure it for a real production service. 1. Liveness Probe: Is the Container Alive? The liveness probe answers one question: "Is this container still running correctly?" If the probe fails, kubelet kills the container and restarts it. livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 failureThreshold: 3 Use liveness probes for deadlock detection . If your app enters a state where it's alive but not making progress (a goroutine leak, a stuck mutex, an infinite loop), the liveness probe exposes that and triggers a restart. The #1 mistake people make: using the liveness probe to check external dependencies like databases or upstream APIs. Don't do this. If your database is down and your liveness probe fails, Kubernetes will restart your pod — but the database is still down. Restarting the app doesn't help, and now you have a crash loop on top of a DB outage. That's worse. Liveness probes should only check internal process health. Not database connectivity, not Redis, not upstream services. 2. Readiness Probe: Is the Container Ready for Traffic? The readiness probe answers: "Should this pod receive traffic?" If it fails, the pod is removed from all Service endpoints. It is not restarted. readinessProbe: httpGet: path: /ready port: 8080 periodSeconds: 5 failureThreshold: 2 successThre

A previous post covered how to absorb PHP 8.2 Deprecated warnings from WP-CLI using a three-layer defense . The approach — prepending WP_CLI_PHP_ARGS to set error_reporting — works in many environments. But a case came up where Deprecated warnings wouldn’t disappear despite the same configuration. Tracing the cause revealed a structural reason why the environment variable never arrived. This post records that root cause and the three-part fix added in v1.6.8. Why environment variables don’t arrive — the phar + shebang execution path An agency reported that on Xserver, plugin list retrieval was failing across multiple sites (referred to here as "site A / site B") with a large volume of Deprecated messages. We reproduced the same behavior on our own Xserver setup (PHP 8.2.30, WP-CLI 2.7.1) and traced the execution path. Xserver’s /usr/bin/wp is a phar binary. Inside, it starts with a #!/usr/bin/env php shebang, so the actual startup sequence looks like this: shell → /usr/bin/wp (shebang: #!/usr/bin/env php) ↓ env locates php and starts it ↓ php loads the phar → WP-CLI runs In this path, WP_CLI_PHP_ARGS is never read as a PHP startup option. WP_CLI_PHP_ARGS is supposed to let WP-CLI pass a -d flag to PHP, but when PHP itself is launched via shebang, control never reaches the point where WP-CLI can inject that flag into PHP’s invocation. # doesn&#8217;t work — /usr/bin/wp on Xserver is a shebang-launched phar WP_CLI_PHP_ARGS = "-d error_reporting='E_ALL & ~E_DEPRECATED'" wp plugin list --format = json # works — -d goes directly to the php binary php -d error_reporting = 'E_ALL & ~E_DEPRECATED & ~E_USER_DEPRECATED' /tmp/wp-cli-2.7.1.phar plugin list --format = json We verified this against our production setup: with the first form, 407 Deprecated lines remained; with the second, 0. Pillar A — detecting the php-direct path and injecting -d The fix: inspect wp_cli_path for whether it’s a php-direct invocation, and if so, inject -d error_reporting immediately after the PHP

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry . This is our official submission for the DEV Big Summer Bug Smash challenge under the #bugsmash track. Below is the technical tale of how we isolated, debugged, and optimized cross-layer node latency issues when deploying our Web3 framework on Polygon. The Problem: The Post-Hard Fork RPC Latency Wall 🐛 During heavy network volume spikes or directly following major ledger upgrades, our automated event listener logging pipeline kept crashing with random, non-deterministic invalid block range exceptions when attempting to pull historical data blocks via standard eth_getLogs routines. The Technical Root Cause The root bottleneck came down to an internal desync inside shared public RPC telemetry environments: The Bor Layer mints new block headers at a blistering speed (~2 seconds). The Internal Indexer DB takes slightly longer to completely unpack, parse, and commit transaction event logs to disk. When our asynchronous scripts called the node, latest grabbed the bleeding edge tip of the chain from memory, but a simultaneous getLogs query hit the slower indexer database. This split-millisecond race condition threw immediate pipeline errors. The Fix: Layered Application Buffering 🛠️ To smash this bug without modifying low-level node client builds, we engineered a programmatic block-padding delay loop directly into our interaction routers. Instead of tracking unfinalized tip block states blindly, we forced our queries to target safe block ranges sitting securely just behind the tip of the chain. // Localized block-buffer deployment fix const currentChainTip = await provider . getBlockNumber (); const indexedBlockBoundary = currentChainTip - 3 ; // Buffer 3 blocks (~6 second safety zone) const targetLogs = await contract . getLogs ({ fromBlock : indexedBlockBoundary - 20 , toBlock : indexedBlockBoundary }); This structural adjustment completely stabilized our off-chain reward data pipeline, gua

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",