标签：#m

Introduction: When Does On-Premises Outpace the Cloud? For small businesses like ComputeLabs , the decision between on-premises servers and cloud services isn’t just about cost—it’s about predictable stability versus elastic flexibility. With stable workloads (websites, email, file storage, backups, internal apps), the question narrows: Does a one-time server purchase amortized over 5–7 years beat monthly cloud bills? The answer hinges on a total cost of ownership (TCO) analysis , where upfront CAPEX collides with recurring OPEX , and hidden costs lurk in both models. The CAPEX vs. OPEX Tug-of-War On-premises servers demand a high initial investment —hardware, software licenses, setup. For a small business, this could mean $5,000–$15,000 upfront , depending on specs. Cloud services, in contrast, operate on a pay-as-you-go model , with monthly costs averaging $100–$500 for similar workloads. But here’s the catch: Cloud costs compound. Over 5 years, that’s $6,000–$30,000 —potentially double the on-premises CAPEX. The break-even point? When the cumulative cloud spend exceeds the depreciated server cost , typically 3–4 years in , assuming no major upgrades. Hidden Costs: The Silent Budget Killers On-premises servers aren’t just a one-time buy. Electricity (a 2U server consumes ~ 500W/hour , costing ~ $400/year ), cooling (fans degrade, heat expands components, shortening lifespan), and maintenance (disk failures, OS patches) add $500–$1,000/year. Cloud services mask these costs but introduce their own: data egress fees (AWS charges $0.09/GB for outbound transfers), premium support ( $100+/month ), and vendor lock-in (migrating data is costly). The edge case? Regulatory compliance —if data must stay on-premises, cloud costs become irrelevant, but self-managed security (firewalls, patches) becomes a non-negotiable expense. Scalability vs. Stability: The Workload Paradox Cloud’s elasticity is its strength—but for stable workloads, it’s overkill. An on-premises server sized

The single hardest thing about getting paid isn't writing the invoice. It's the follow-up — knowing when to nudge a quiet client, and doing it in a tone that doesn't torch the relationship. Most tools solve this with a dumb cron job: "send a reminder 7 days after the due date." That's wrong for almost everyone, and here's why. The problem with fixed reminder schedules Payment behavior isn't uniform. One client pays like clockwork on day 32 of a "net 30" invoice — not late, just their rhythm. Another pays on day 5 but only if you remind them on day 3. A blanket "day 7 past due" reminder annoys the first client (who was always going to pay) and misses the second (who needed the poke earlier). So the real problem is per-client timing prediction , not scheduling. You want to model each client's payment distribution and act at the point where a reminder has the highest marginal effect — the moment they're most likely to convert intent into a transfer. Modeling payment rhythm as a per-client distribution Every invoice gives you a labeled data point: (sent_date, due_date, paid_date, amount, was_reminded) . Over time, per client, that's a distribution of "days from send to pay." The naive move is to average it. Don't — averages hide the shape, and the shape is the whole signal. We model each client's pay-day as a distribution and track two things that matter more than the mean: Dispersion — a tight distribution (always day 30–32) means a reminder before day 30 is noise. A wide one means the client is reminder-sensitive. Reminder lift — comparing paid-day distributions with and without a nudge tells you whether reminders actually move this client, and by how much. for client in clients : hist = paid_events ( client ) # list of days-to-pay p50 , p90 = quantiles ( hist , [. 5 , . 9 ]) lift = mean ( days_without_reminder ) - mean ( days_with_reminder ) # act just before the client's own habitual pay point, # but only if a nudge historically helps them if lift > MIN_LIFT_DAYS :

If someone compromised one of your project's dependencies today, would they be able to steal your...

🎲 A Tale of Inference Woes 🎲 Thy context window overflows with dread, Thy API keys scattered 'cross thy thread. Thou switchest providers mid-conversation, And pray thy tokens find the right foundation. No ward stands guard when tools go rogue, No grimoire saves a session from the fog. Thy agents wander, masterless and blind, Thy prompts untested—leaving truth behind. Thy wallet weeps. Thy latency doth creep. Thy model's fine. Thy infrastructure? Not so deep. Sound familiar? I'm excited to share the public README for Arcanum — a .NET 10, single-binary, Native AOT, local-first AI inference hub that treats your infrastructure with the seriousness of a dungeon master and the organization of a well-kept grimoire. Arcanum is one self-contained native executable. No runtime prerequisite. No "install the framework first." Just arcanum serve and you're running a full inference platform on loopback. What's in the bag of holding: 🏰 Local-first & encrypted — SQLCipher-encrypted Grimoire persists every session, entry, and memory. Your data never leaves your machine unless you tell it to. ⚔️ Multi-provider native engine — Any OpenAI-compatible API (DeepSeek, Groq, Ollama via /v1, LM Studio, etc.) plus local GGUF models via a managed llama-server lifecycle. One hub, zero vendor lock-in. 🔮 OpenAI API compatible — POST /v1/chat/completions and GET /v1/models work with existing OpenAI clients out of the box. Drop-in replacement for your local stack. 🛡️ Wards & Sanctum — High-risk tools require operator approval before execution. Per-campaign sandboxes enforce path containment, network policy, and OS-level CPU/memory/FD limits via cgroups v2 and setrlimit. 📜 Spells, not prompts — Versioned markdown workflows with dependency resolution, tool allowlisting, and semantic routing. Dry-run cast previews before spending a single token. 🧙 Autonomous Apprentices — Goal-driven agents with plan generation, retry/backoff, autonomous plan revision, DM escalation, and parallel step execution. 🏰 The

Most SaaS backends were built around a simple assumption: The user pays a subscription, then uses the product. That assumption breaks down for AI apps. An AI app does not just serve screens. It spends money while it works. A user searches the web. A model summarizes a report. An image model generates a draft. An agent calls an MCP tool. A workflow buys data from an API. A future x402 endpoint charges for a capability call. Every one of those actions can have a marginal cost. That means the backend for an AI app is no longer just a place to store users, projects, and settings. Increasingly, it is a system of record for economic activity. In other words: AI app backends are becoming accounting systems. The old SaaS model was simpler Traditional SaaS could survive with coarse billing. You had: monthly subscriptions seats tiers maybe a usage limit somewhere That worked because the marginal cost of most product actions was close enough to zero. If a user clicked a button, edited a document, opened a dashboard, or created a project, the backend cost was usually small compared with the subscription price. The business could average it out. AI apps are different. The product may call paid APIs on almost every useful action. Search once. Summarize once. Generate once. Transcribe once. Call an agent tool once. The unit economics are inside the interaction loop. If the product cannot see who spent what, when, and why, the business is flying blind. Usage billing is not an add-on For AI apps, usage billing is often treated like a pricing feature. I think that is too narrow. Usage billing is really a cost ledger. It answers: which user triggered the cost? which project or app did it belong to? which capability was called? what did it quote before execution? what did it actually cost? was it retried? was it idempotent? did the end user pay for it? is there a payment or checkout record attached? If you cannot answer those questions, you do not have a reliable production backend for

We’ve all been there: it’s 7:00 PM, you’re exhausted after a long sprint, and you open a food delivery app. Your brain screams "Double Cheeseburger," but your body is still recovering from that mid-afternoon sugar spike. What if your phone was smart enough to say, "Hey, your blood sugar is currently 160 mg/dL and rising—maybe skip the extra fries?" In this tutorial, we are building a Chief Health Officer (CHO) Agent . This isn't just a simple chatbot; it’s a sophisticated AI Agent using LangGraph to bridge the gap between real-time medical data (CGM) and real-world actions (Food Delivery APIs). By leveraging automation , function calling , and state machines , we’ll create a system that actively protects your metabolic health. The Architecture: How the CHO Agent Thinks To build a reliable agent, we need a "stateful" workflow. We aren't just sending a prompt to an LLM; we are creating a loop that monitors glucose levels, analyzes food options, and interacts with the browser. graph TD A[Start: Hunger Trigger] --> B{Fetch CGM Data} B -->|Sugar High/Unstable| C[Constraint: Low GI Only] B -->|Sugar Stable| D[Constraint: Balanced Meal] C --> E[Scrape Delivery App Menu] D --> E E --> F[Agent: Analyze Ingredients & GI Index] F --> G[Selenium: Mark/Filter Non-Compliant Items] G --> H[End: Safe Ordering] subgraph "The LangGraph Loop" C D E F end Prerequisites Before we dive into the code, ensure you have the following: LangGraph & LangChain : For the agent's cognitive architecture. Dexcom API Credentials : To fetch real-time Continuous Glucose Monitor (CGM) data. Selenium : For interacting with food delivery web interfaces (Meituan/Ele.me). OpenAI API Key : Specifically for GPT-4o’s reasoning and function-calling capabilities. Step 1: Defining the Agent State In LangGraph, everything revolves around the State . Our CHO agent needs to track the current glucose level, the user's health constraints, and the list of available food items. from typing import TypedDict , List , Anno

I built a 36-pattern checklist to catch AI writing tells in my own drafts, calibrated against...

If your memory is hosted, your thoughts are leased. We did not just move our files to the cloud. We moved our working memory. Andy Clark and David Chalmers called it the extended mind in 1998. The thesis was simple. Cognition leaks into the tools we trust. A notebook can be part of your mind if you access it reliably. In 2026, that notebook is a vector database owned by a platform with a legal department. Your extended mind now has terms of service, retention policies, and a compliance team that answers subpoenas faster than you answer email. I am not interested in nostalgia for paper. I am interested in architecture that preserves agency. The fix is not to think less with machines. It is to think locally with machines you control. That is why I built a second brain that lives on a Raspberry Pi 5 with NVMe and a Hailo-8 accelerator, running Retrieval Augmented Generation completely offline. No API keys. No telemetry. No third party that can be compelled to hand over your associative graph. This is the expanded blueprint. More cohesive, more rigorous, and more useful than the usual cloud versus local sermon. The extended mind, now with a landlord The original extended mind argument was about trust and coupling. If you reach for a tool as automatically as you reach for a memory, it counts as cognition. The cloud broke that coupling by inserting a landlord. Your retrieval is fast, but it is also observed, logged, ranked, and retained. Three consequences follow. First, epistemic pollution. When your queries train their models, your future answers are shaped by everyone else’s queries. Your private context gets diluted by the median user. Second, legal exposure. Your prompts, your uploads, your retrieval history, and your embeddings are business records. In many jurisdictions they are discoverable. You cannot plead the fifth for data you gave to a provider. Third, strategic fragility. A policy change, a price hike, a region block, and your cognitive prosthesis goes dark.

Back in February, Uber announced ambitious plans to launch in seven new European markets in 2026 — but now five of those launches are reportedly on hold.

Here's the failure mode I keep running into. A team gives a coding agent a repo, a task, and maybe a README. The agent can find files and write code, but it still has to guess the operating rules. It guesses the package manager. It guesses which checks matter. It guesses whether generated files are safe to edit. It guesses what "done" means. A README is usually for humans: what the project is, how to run it, and where the important docs live. A coding agent needs different context. Setup rules. Test commands. Boundaries. Completion criteria. That's the gap AGENTS.md fills. The official AGENTS.md guidance describes it as a predictable place for coding-agent instructions: setup commands, test commands, code style, security considerations, and nested instructions for large monorepos. I find the split useful in a more boring way. The README answers, "What is this project?" AGENTS.md answers, "What should an agent know before touching it?" That second question is where the work usually gets fragile. Where Goose Fits Goose makes this less theoretical because it isn't just a chat box. It's an open source local AI agent with a desktop app, CLI, API, MCP extensions, and skills. Without AGENTS.md , I find myself writing prompts like this: Update the docs, but don't touch generated files, use pnpm, run the lint and test commands, keep the PR small, and tell me what you couldn't verify. With AGENTS.md , the prompt can get shorter: Update the quickstart docs for the new config flag. Goose can run the task in the repo. The repo can carry the standing instructions. I noticed this on a small docs/config update where generated files sat near source files. Without repo instructions, the prompt had to carry the package manager, generated-file boundary, checks, and the "tell me what you could not verify" rule. Once those rules lived in AGENTS.md , the prompt became just the task. Not magic. Just fewer chances to forget the boring parts. Where Skills Fit I would add one more layer once

# Sakana Fugu: The Multi-Agent AI System That Works Like a Team We’ve all been there: copy-pasting a prompt from ChatGPT to Claude, and then to Gemini, trying to find which AI gives the best answer. Different AI models have different strengths. Some are excellent programmers, some write beautifully, and others are master logicians. But constantly switching between them is slow and frustrating. Sakana AI has introduced a brilliant solution: Sakana Fugu . Fugu is a multi-agent AI system that bundles multiple frontier models into a single, seamless package. To you, it looks like a single chatbot. But behind the scenes, it acts as a project manager, coordinating a team of top-tier AI models to solve your task. What is Sakana Fugu? Sakana Fugu is a collaborative AI framework. Instead of relying on one massive AI model to do all the work, Fugu orchestrates a pool of specialized models (such as GPT-5, Claude Opus, and Gemini Pro). When you give Fugu a complex, multi-step prompt, it: Analyzes the task and breaks it down into steps. Assigns specialized roles (like researcher, coder, and editor) to different AI models. Passes the work back and forth between them until the final, polished result is ready. By working as a team, these models achieve far better results than any single AI could on its own. The Secret Sauce: Teamwork Over Size Fugu’s intelligent coordination is based on two core concepts presented at the ICLR 2026 conference: 1. The Manager (TRINITY) Think of TRINITY as the manager of the team. It is a compact coordinator model that assigns specific roles to the worker LLMs. For example, it might tell Gemini to generate the initial code, tell Claude to find the bugs, and tell GPT to write the user documentation. They collaborate seamlessly without needing to merge their code bases. 2. The Conductor The Conductor acts as the communication network designer. It figures out the best way for the worker AIs to talk to each other for your specific question. It writes cust

One of the most important ideas in modern wireless communication did not come out of a corporate research lab or a defense contractor. It was patented in 1942 by one of the most famous movie stars of the era, working alongside an avant-garde composer. The actress was Hedy Lamarr, and the technique she helped invent, frequency hopping , is a direct ancestor of the Wi-Fi, Bluetooth, and GPS signals your devices rely on every day. The patent Hollywood forgot At the height of her Hollywood fame, Hedy Lamarr was also a self-taught inventor who tinkered between film shoots. Early in World War II she became fixed on a hard problem: radio-controlled torpedoes were easy to jam, because an enemy who found the single control frequency could simply drown it in noise and send the weapon off course. Working with composer George Antheil, she designed a system where the transmitter and receiver would rapidly and secretly switch together across many different frequencies. Antheil, who had once synchronized sixteen player pianos for a concert piece, suggested using a slotted paper roll like a player piano to keep both ends hopping in step across 88 frequencies , the same number as the keys on a piano. On August 11, 1942, they received U.S. Patent 2,292,387 for a "Secret Communication System." The U.S. Navy filed the idea away and did not use it during the war. For decades the patent sat largely forgotten, and Lamarr received no money and little recognition for it in her lifetime. She was finally inducted into the National Inventors Hall of Fame in 2014, years after her death. What frequency hopping actually does The core insight is deceptively simple. Instead of putting a signal on one fixed frequency, you spread it across many frequencies in a pattern that only the sender and receiver know. Both ends "hop" in perfect synchronization, dwelling on each frequency for only a fraction of a second before jumping to the next. This buys you two enormous advantages. It is very hard to jam, b

Nearly 1 million people have lost a total of $3.8 billion after buying President Donald Trump’s $TRUMP memecoin, while Trump made $636 million.

🎮 I Built a Free Browser Gaming Platform – FizGame Over the past few weeks, I've been working on a side project called FizGame, a free browser gaming platform where anyone can instantly play HTML5 games without downloads or installations. 🌐 Website: FizGame Why I Built It I wanted to create a simple platform where players can: 🎮 Play instantly in their browser 🚫 No downloads or installations 📱 Play on both desktop and mobile ⚡ Fast loading experience Current Features Hundreds of HTML5 games Mobile-friendly design Instant game loading Categories like Puzzle, Action, Arcade, Racing, and Casual Search and game discovery Responsive UI Tech Stack PHP Symfony MySQL HTML5 Games JavaScript CSS Nginx Current Focus I'm currently working on: Better SEO Faster page speed AI-generated game descriptions Improved game recommendations Daily game publishing Social media automation Biggest Challenge One of the hardest parts isn't building the website—it's helping people discover it. I'm experimenting with: YouTube Shorts Instagram Reels Facebook Reels Pinterest Organic SEO If you've built a gaming website before, I'd love to hear what worked best for you. Feedback Welcome I'd appreciate any feedback on: UI/UX Performance Navigation SEO Features you'd like to see 🎮 Check it out: FizGame Thanks for reading! 🚀

PostgreSQL provides several planner configuration parameters, such as enable_seqscan and enable_indexscan , that influence how execution plans are generated. These settings affect planning, not the execution of an already-generated plan. With prepared statements, this raises an interesting question. Should planner settings be applied before PREPARE, before EXECUTE, or both? Let's look at a simple example: a "tasks" table with a due date and a "done" status: \ c drop table if exists tasks ; -- a table of tasks with status (done or not) and due date create table tasks ( id bigint generated always as identity primary key , due timestamptz , done boolean ); -- insert 500 tasks, with 1% not done insert into tasks ( due , done ) select now () + interval '1 day' * n , 42 != n % 100 from generate_series ( 1 , 500 ) n ; -- index the todo (partial index) create index on tasks ( due , id ) where done = false ; vacuum analyze tasks ; With a partial index, I indexed only the tasks that are not yet done ( done = false ) because that's my most frequent query pattern: postgres =# explain select id , due , done from tasks where done = false and id > 0 order by due limit 1 ; QUERY PLAN --------------------------------------------------------------------------------------- Limit ( cost = 0 . 13 .. 3 . 60 rows = 1 width = 17 ) -> Index Scan using tasks_due_id_idx1 on tasks ( cost = 0 . 13 .. 17 . 47 rows = 5 width = 17 ) Index Cond : ( id > 0 ) ( 3 rows ) With partial indexes, the condition covered by the index is not even visible in the execution plan because the index itself enforces the condition. Prepared statement I decided to use a prepared statement with all values as parameters. It is probably not a good idea in this case. When a parameter can have only a few different values and you expect different cardinalities for each, you should probably define one query per value, using literals. I'm doing this to illustrate what can happen, with a simple, extreme example: postgres =# pr

Introduction Every post in this series has quietly touched a piece of the same problem. Building Agentic Workflows in Java said toolUse.input() is untrusted and must be validated before it reaches your code. Building Reliable LLM Applications in Java said the model will confidently invent facts, so ground it and get typed output instead of parsing prose. Neither post named the thing underneath both statements: anything that crosses from outside your code into the model, or from the model back into your code, is untrusted input — a request body from the network, not a trusted internal value. This post names that boundary directly and gathers the defenses in one place — prompt injection (direct and indirect), input validation, output validation, and PII redaction — with the SAFE pattern shown beside every unsafe one it replaces, since this is the security-forward capstone of the series. The Trust Boundary: Three Kinds of Untrusted Input An LLM application has three places where untrusted text enters: User input — anything a person types, uploads, or submits through an API. Retrieved content — Making RAG Accurate in Java built a pipeline that ranks and returns chunks from a document store; those chunks were written by whoever authored the source document, not by you, and a malicious or compromised document can carry text aimed at the model reading it, not at a human reader. Model output — untrusted the moment it's about to be used rather than displayed : passed to a tool, interpolated into a query, or fed into another LLM call as context. A model that just read attacker-controlled retrieved text can be manipulated into producing attacker-controlled output. The single rule under all three: text is data until your code has explicitly decided it's safe to use for anything more than display. Nothing below is executed against a live API — every snippet is illustrative, and none of it uses a real key or a real record. Direct Prompt Injection: Defending the System Prompt A di

Introduction Every post in this series has quietly touched a piece of the same problem. Building Agentic Workflows in Python said a tool's input is untrusted and must be validated before it reaches your code. Building Reliable LLM Applications in Python said the model will confidently invent facts, so ground it and get typed output instead of parsing prose. Neither post named the thing underneath both statements: anything that crosses from outside your code into the model, or from the model back into your code, is untrusted input — a request body from the network, not a trusted internal value. This post names that boundary directly and gathers the defenses in one place — prompt injection (direct and indirect), input validation, output validation, and PII redaction — with the SAFE pattern shown beside every unsafe one it replaces, since this is the security-forward capstone of the series. The Trust Boundary: Three Kinds of Untrusted Input An LLM application has three places where untrusted text enters: User input — anything a person types, uploads, or submits through an API. Retrieved content — Making RAG Accurate in Python built a pipeline that ranks and returns chunks from a document store; those chunks were written by whoever authored the source document, not by you, and a malicious or compromised document can carry text aimed at the model reading it, not at a human reader. Model output — untrusted the moment it's about to be used rather than displayed : passed to a tool, interpolated into a query, or fed into another LLM call as context. A model that just read attacker-controlled retrieved text can be manipulated into producing attacker-controlled output. The single rule under all three: text is data until your code has explicitly decided it's safe to use for anything more than display. Nothing below is executed against a live API — every snippet is illustrative, and none of it uses a real key or a real record. Direct Prompt Injection: Defending the System Prompt

Introduction We already covered picking the right model tier for the task and caching a large shared prefix in https://pg-blogs.netlify.app/posts/11-building-reliable-llm-apps-in-java/ . Those two lines were the tip of a bigger discipline: LLM cost is not a fixed line item, it's an engineering variable — one you can measure and shrink with the same rigor you'd apply to database query time or container memory. This post goes deeper: how input/output pricing actually works, the exact cache_control shape and how to prove a cache hit rather than assume one, the Batches API for work that isn't latency-sensitive, and model routing — using a cheap model to triage, escalating only the hard cases to a stronger one. The honest framing throughout: measure before you optimize. Every technique here has a cost of its own; applied to the wrong workload, "optimization" makes things slower or more expensive. Token Economics: Why the Prefix Is the Bill Anthropic (like every hosted LLM provider) prices input and output tokens separately, and output is always pricier — the model has to generate output autoregressively, one token informed by all the ones before it, while input can be processed in parallel. Representative pricing from the current model catalog: Model Input Output Claude Opus 4.8 $5.00 / MTok $25.00 / MTok Claude Sonnet 5 $3.00 / MTok $15.00 / MTok Claude Haiku 4.5 $1.00 / MTok $5.00 / MTok Two consequences follow directly: Long system prompts, tool definitions, and RAG context are read on every request , not written once. A 20K-token system prompt sent on every one of 10,000 requests is 200M input tokens — at Opus 4.8 rates, $1,000 before a single output token is generated. The shared prefix , not the user's question, is usually where the money goes. A verbose model wastes money twice — once on the extra output tokens themselves, and again because the next turn's messages history now carries that verbosity forward as input on every subsequent call. Trimming max_tokens an

Introduction https://pg-blogs.netlify.app/posts/10-building-reliable-llm-apps-in-python/ closed with a section on picking the right model per task and caching a shared prefix. That was the entry point into a bigger discipline: LLM spend is an engineering variable, not a fixed bill — one you can measure and reduce with the same rigor you'd apply to query latency or memory footprint. This post goes deeper on four levers: how input/output pricing actually works and why the prefix is usually where the money goes, the exact cache_control shape and how to prove a cache hit instead of assuming one, the Batches API for work that isn't latency-sensitive, and model routing — a cheap model triaging requests and escalating only the hard ones. The throughline is honest: measure before you optimize. Every lever here has its own cost; misapplied, it makes things slower or pricier, not cheaper. Token Economics: Why the Prefix Is the Bill LLM providers price input and output tokens separately, and output always costs more — generation is autoregressive (each token depends on every one before it), while input can be processed in parallel. Representative pricing from the current model catalog: Model Input Output Claude Opus 4.8 $5.00 / MTok $25.00 / MTok Claude Sonnet 5 $3.00 / MTok $15.00 / MTok Claude Haiku 4.5 $1.00 / MTok $5.00 / MTok Two things follow: A long system prompt, tool list, or RAG context is billed as input on every request , not written once. Send a 20K-token system prompt on 10,000 requests and that's 200M input tokens — at Opus 4.8 rates, $1,000 before the model has generated a single output token. The shared prefix , not the user's actual question, is usually the dominant cost. Verbose output costs twice — once directly (more output tokens billed at the higher rate), and again because the next turn's history carries that verbosity forward as input. Asking for concise output and setting a sane max_tokens is a cost control, not just a style choice. This is why the tw

Introduction Building Reliable LLM Applications in Python put it plainly: treat model output as a hypothesis to verify, not a fact to trust. Testing Best Practices in Python put the same discipline in pytest terms: a suite only earns trust by asserting the right things at the right level, unhappy paths included. This post is where those two ideas meet — a pytest assertion either passes or fails against a fixed expected value; an LLM's output is a paragraph of prose that might be right in spirit while differing token-for-token from anything you wrote down in advance. Evaluating it takes a harness, not an assert . That harness has three parts: a golden dataset of representative cases with known-good expected behavior, scoring that turns each case into a pass/fail or a number, and regression testing that runs the harness on every change and fails the build when the score drops. Making RAG Accurate in Python already gave you half of this story — recall@k, precision@k, MRR, nDCG measure whether retrieval found the right chunks. This post measures the other half: whether the generated answer built from those chunks is actually good, which is a genuinely different question a retrieval metric can't answer on its own. Everything below is illustrative, non-executed Python, grounded in the same Anthropic SDK shapes as posts 10/11. The Golden Dataset: Curating Cases, Not Just Inputs A golden dataset is a small, hand-curated set of (input, expected behavior) pairs that represents the ways your application is actually used — not a random sample, and not just the cases that already work. Each case needs enough structure to be scored automatically later: from dataclasses import dataclass , field @dataclass class EvalCase : id : str category : str # "extraction", "qa", "summarization", ... input : str # the prompt/question sent to the system under test expected_exact : str | None = None # non-None only for cases scorable by exact match must_contain : list [ str ] = field ( default_f