标签：#LLM

Your LLM bill is exploding because 70% of user queries are semantically identical, yet your traditional cache ignores them completely. Even worse, if you implement semantic caching poorly, a single bad actor can poison your entire AI model's knowledge base, leading to incorrect or malicious responses for legitimate users. The Cost of Redundancy in LLM Systems Imagine running an AI-powered customer support chatbot for an e-commerce platform. Users frequently ask things like, "What's your return policy?", "How can I send this item back?", or "Do you offer refunds if I'm not satisfied?". To an LLM, these are distinct prompts, each triggering an expensive API call to OpenAI or Anthropic, costing you dollars per thousand tokens. On the surface, it looks like individual requests. But structurally, they all ask the same question with a similar intent. Your traditional HTTP cache, which relies on exact string matches, sees "What's your return policy?" and "How can I send this item back?" as entirely different requests. It misses the semantic similarity. So, for every variation of the same question, you're making a full LLM inference call. If 50-70% of your user queries fall into these semantically redundant categories, your LLM costs skyrocket. For a system handling millions of requests daily, this can quickly turn a profitable product into a money pit, all while adding unnecessary latency for your users. Semantic Caching: The "Fast Path" for LLMs Semantic caching solves this by moving beyond exact string matches. Instead of looking for an identical prompt, it looks for prompts that mean the same thing. It works by converting incoming user prompts into numerical vector representations (embeddings) and then performing a similarity search against a cache of previously embedded prompts and their corresponding LLM responses. Here's the workflow: USER PROMPT | v [ EMBEDDING MODEL ] -- Transform Prompt to Vector (e.g., [0.1, 0.5, -0.2, ...]) | v [ VECTOR DATABASE / CACHE ] | +--

What is an LLM evaluation harness? A deep dive into lm-eval-harness You fine-tuned a 7B model. It aced your smoke tests, your colleague ran a few prompts and shrugged approvingly, and the README is now full of cherry-picked outputs that look great in a screenshot. Then someone asks: how good is it, really? — and you realize you have no number to point at. No MMLU score. No HellaSwag. Nothing reproducible, nothing you can defend in a PR review, nothing you can compare to last week's checkpoint. That's the gap an evaluation harness fills. It turns "vibes-based evaluation" into something with a score, a stderr, and a config file you can re-run next Tuesday. Why evaluate LLMs at all? Two reasons that actually matter: Comparability. If you can't put a number on a model, you can't compare it to anything else — not the previous checkpoint, not the open-source baseline, not the commercial API you're trying to replace. Leaderboards are noisy and gaming-prone, but a local leaderboard with the tasks you care about is one of the most useful artifacts a team can build. Regression detection. Most model regressions are silent. A 0.3-point drop on MMLU won't show up in a chat session, but it will show up in CI. People who ship models for a living treat evals the way backend engineers treat unit tests: mandatory, run on every PR, and blocking on regressions. You don't need a hundred benchmarks. You need the three to five tasks that map to your actual use case , plus one or two general capability anchors (MMLU, HellaSwag) so you can sanity-check that you didn't accidentally destroy basic reasoning while you were tuning for your domain. What is an "evaluation harness"? An evaluation harness is the software that sits between a model and a benchmark. It handles the boring-but-critical parts: loading the model weights, tokenizing prompts in the way the benchmark expects, running inference, extracting the answer from a longer generation, scoring it against a ground-truth key, aggregating

Here's a trace that reset how I think about evaluating tool-calling agents. An agent tries to book a flight. It calls search_flights with departure_date="next Friday" . The endpoint expected an ISO date, so it returns a 400 . The agent retries the same string four times, then apologizes to the user and gives up. Now the part that actually bothered me. Tool selection was correct. The model picked the right function out of a registry of 28. My tool-selection accuracy logged a clean 1.0 . The aggregate task-completion logged a 0 . And neither number told me which of three things broke: the argument was wrong, the model never read the 400 body, or the retry policy looped on the same input. My eval wasn't wrong. It was asking the wrong question. What "tool-call accuracy" actually grades If the only thing you measure is did the agent call the right tool , you're testing intent, not execution. Tool selection is necessary, not sufficient. It passes the moment the right function name shows up in the trace, completely blind to whether the arguments were garbage, whether the model read what came back, or whether it recovered from the 400 . That's the gap. The metric checks that the agent started the right way. Production needs to know whether it finished the right way. The reframe: it's four eval problems, not one The thing I had to internalize is that tool-calling eval is four problems stacked, each with its own root cause: Tool selection , right tool, or correctly no tool Argument extraction , schema-valid and semantically correct Result utilization , did it actually use what the tool returned Error recovery , did it retry, fall back, or escalate Score them separately and "the agent failed" collapses into "the argument extractor regressed on date strings on the flight-booking path." One bisect instead of three days. What I rebuilt Layer 1: Tool selection (with the bucket everyone drops) F1 on the tool name, so a 28-tool registry doesn't hide a regression on one rare endpoint

Hand the same paired before/after dataset (n = 25) to ChatGPT five times. Same prompt: "These are the same subjects measured before and after an intervention. Did their scores change significantly?" Four of the five runs return p = 0.009 from a paired t-test. The fifth run does a Shapiro–Wilk normality check on the differences first, decides they're non-normal, switches to a Wilcoxon signed-rank test, and reports p = 0.000018 . All five reach the same conclusion (significant). But notice what happened: only one run out of five thought to check an assumption you'd want it to check. The other four skipped it. The choice of method — and the test statistic, and the p-value — depended on whether the LLM happened to run an assumption check that time. On borderline data, this is the difference between reject and don't reject. If you're using LLMs for exploratory data analysis on a weekend project, you might shrug. If you're using them for anything that gets cited, gets submitted to a regulator, or gets handed to a clinician, this is a problem. It's a known problem — Cui & Alexander (2026) documented exactly this kind of method-divergence empirically; AIRepr (Zeng et al., 2025) shows the same thing across reproducibility metrics. The current answer in the literature is to constrain the agent so its execution is replayable. But replayability fixes "did we run the same code." It doesn't fix "did we run the right analysis." I've spent the last two months building a different fix. The more interesting half is the architecture. Let me walk through it. The real problem isn't temperature The first reflex is "set temperature=0 ." It's not enough. temperature=0 doesn't make a tool-using agent deterministic across runs. Three reasons: Inference isn't bitwise deterministic, even at temperature=0. Production LLM serving batches requests dynamically, and the attention kernels aren't batch-invariant — so the same input produces different output tokens depending on what other requests it

The Setup I've been building AI agents that use tools — reading files, running commands, calling APIs. There are two main ways to give agents these tools: MCP (Model Context Protocol) — the new standard everyone's adopting Direct CLI calls — good old command-line execution Everyone says MCP is the future. But nobody talks about the token cost . So I measured it. The Test I built a simple file-reading tool and measured the exact token consumption for each approach: Method Tokens per Call Latency (avg) MCP (structured) ~3,400 tokens 280ms CLI + raw output ~200 tokens 45ms Ratio 17x 6x Why MCP Uses So Many Tokens The overhead comes from three places: 1. Tool Schema in Every Request MCP sends the full JSON Schema of every available tool with each request to the LLM. My simple file-reader schema alone is ~800 tokens. With 10+ tools, that's 8,000+ tokens of schema on every single call. { "name" : "read_file" , "description" : "Read contents of a file at given path" , "parameters" : { "type" : "object" , "properties" : { "path" : { "type" : "string" , "description" : "File path to read" } }, "required" : [ "path" ] } } 2. Structured Response Wrapping MCP wraps every response in a structured envelope with metadata, status codes, and typed content blocks. A simple "file not found" error becomes a 200-token JSON object. 3. Round-Trip Protocol Overhead Each MCP call involves: request → server parse → execute → format response → return → client parse → extract. Each step adds tokens for protocol framing. The CLI Alternative With direct CLI execution: $ cat /path/to/file.txt [ raw file content] That's it. Raw input, raw output. No schemas, no envelopes, no metadata. When MCP Is Worth It Despite the token cost, MCP shines when: You need standardized discovery — agents dynamically finding available tools You're building reusable tool servers — one MCP server serves many agents Security sandboxing matters — MCP's permission model is more granular Team collaboration — shared tool de

This is the technical, reproducible version of a fix I shipped on my own homelab. If you want the narrative version, that's on Medium. This one is the recipe: the measurements, the math, the Modelfile, and the exact prompt I gave Claude Code to generate it. Copy-paste friendly. Repo for the dashboard used throughout: https://github.com/SikamikanikoBG/homelab-monitor TL;DR One 24GB RTX 3090, two GPU services: WhisperX large-v3 (STT, 7.7GB peak) and a Devstral Small 24B email-triage LLM (Q4_K_M, ~18.3GB). 18.3 + 7.7 = 26GB → CUDA OOM whenever they overlapped. The LLM was loaded with a 40k context window but the triage job never needed more than ~5–8k tokens. Capped num_ctx to 8192 → KV cache drops from ~6.1GB to ~1.25GB → model footprint ~18.3GB → ~14.2GB . 14.2 + 7.7 = 21.9GB → both resident, zero OOM, no quality loss. The setup Host : openSUSE, Xeon (56 threads), 125GB RAM, 1x RTX 3090 (24GB) GPU svc : WhisperX large-v3 (speech-to-text) GPU svc : Ollama -> devstral-small-2 (24B, Q4_K_M) for background email triage Both services run all the time. The OOM only happened when I dictated to my assistant (WhisperX) while the triage loop was active. Step 1 — Make the contention measurable nvidia-smi shows instantaneous VRAM. It can't show you which service spiked or when two of them overlapped — and an intermittent OOM is a timing problem. You need per-service VRAM history. I use my own dashboard (homelab-monitor) for this. The relevant view is "AI Models", which attributes VRAM per model server and per loaded model, over a time range, with OOM markers and a capacity ceiling line. What the history showed at the overlap window: Service Peak VRAM Devstral 24B (triage) ~18.3 GB WhisperX large-v3 7.7 GB Total ~26 GB on a 24 GB card If you want to reproduce the measurement, the dashboard runs as a single container: git clone https://github.com/SikamikanikoBG/homelab-monitor cd homelab-monitor docker compose up -d --build # open http://<host>:9800 -> AI Models / GPU views (NVIDI

The Model Comparison Problem Most AI model comparisons are useless for developers making real decisions. They benchmark on academic datasets that don't reflect production workloads. They test frontier capabilities that matter for 5% of use cases. They ignore latency, cost, rate limits, and API reliability — which are the things that actually determine whether a model works in your application. This comparison is different. It's focused on what matters when you're building something: how Grok and Gemini perform on the types of tasks developers actually encounter, what each model's API experience is like, and where the genuine tradeoffs lie. I'm deliberately not including benchmark scores. If you want MMLU numbers, there are plenty of leaderboards for that. This is about production utility. What Each Model Actually Is Grok (xAI) Grok is xAI's model family. The current production models are Grok-3 and Grok-3 Mini, with Grok-3 being the flagship. Grok has a large context window (128K tokens standard, with extended context available), real-time access to X (Twitter) data as a differentiating feature, and strong performance on reasoning-heavy tasks. The xAI API follows a familiar REST pattern and is broadly compatible with OpenAI SDK conventions, which makes migration straightforward. Grok's notable characteristics: Strong at structured reasoning and multi-step problem decomposition Real-time web access via the API (useful for tasks needing current information) Relatively generous rate limits compared to some competitors Less restrictive on certain content categories than some other models Gemini (Google DeepMind) Gemini is Google's model family, currently anchored by Gemini 1.5 Pro and Gemini 2.0 Flash. The defining feature of Gemini is its context window — Gemini 1.5 Pro supports up to 1 million tokens in production, which is genuinely useful for certain document-heavy use cases. Gemini also has the tightest integration with Google's ecosystem (Workspace, Cloud, Search)

Claude Opus 4.8 shipped today. The benchmarks are a distraction — here is what actually changes about how your agents run tomorrow. Anthropic announced Claude Opus 4.8 at 16:00 UTC on June 3, 2026. The launch post leads with the usual benchmark deltas: SWE-bench Verified up 4.1 points, GPQA Diamond up 2.9, TAU-bench tool-use up 6.4. There is a chart. There is a marketing line about "the most capable agentic model we have ever shipped." If you stop reading there, you will miss the three things that will change how your production agents behave starting tomorrow. I have spent the morning re-running our internal agent harness against Opus 4.8 and reading the model card line by line. Two of the three changes are improvements. One of them is a silent regression that will bite anyone who pinned the model ID. Here is the full picture. What 4.8 actually changes The model card and release notes ship three changes that the launch blog post does not foreground: Cache-aware routing inside long agentic loops. The 4.7 router treated every tool-call cycle as a fresh planning step. 4.8 keeps an internal trace of which cache breakpoints were hit on the previous step and biases the next plan toward extending those traces. In agent harnesses that already use prompt caching aggressively (Claude Code, the Agent SDK with cacheControl: "ephemeral" on the system prompt), cache hit rates jumped from a measured ~46% on 4.7 to ~71% on 4.8 across a 30-step coding loop. The 200k context window now actually behaves at 200k. Anthropic published a needle-in-a-haystack chart in the model card going out to 200,000 tokens. The 4.7 chart got noticeably worse past ~140k tokens; the 4.8 chart is flat. This sounds like a benchmark thing. It is not. It changes the cost equation for "just stuff everything in context" patterns that 4.7 quietly punished by degrading accuracy. claude-opus-4-7 was not aliased. The launch shipped a new model ID — claude-opus-4-8 — and the previous ID is still callable. But if y

Since the early days of GenAI, when ChatGPT launched in late 2022, we began using prompt engineering to direct chatbots (and later LLMs) with human language instructions to provide us answers to questions or take actions (in a high-level…) In 2025, companies such as OpenAI and Anthropic began releasing a new agentic concept called “AI Agent”, an autonomous system that uses an AI model as its "brain" to perceive an environment, make independent decisions, and execute multi-step tasks using digital tools. Unlike passive chatbots that just answer questions, an agent can plan its own workflow, run commands, and browse the web to achieve a specific goal without constant human supervision. In this blog post, I will explain the concept of Context-as-Code and share some coding examples. Introducing Context-as-Code Traditional prompting is a one-way street. You type out your instructions, send them off, and that text never changes. AI agents operate completely differently. Because they work on their own, every action they take creates a mountain of new data. Every time an agent opens a file, checks an error, or runs a tool, it adds more information to the pile, which quickly overwhelms a standard chat screen. Context-as-Code treats the agent like a stateless compute engine. Instead of a massive text prompt, we use version-controlled files ( CLAUDE.md , AGENTS.md ) to establish structural boundaries, separating the permanent project rules from the temporary, dynamic session memory. Context-as-Code transforms loose AI prompts into version-controlled engineering assets by using structured Markdown files to establish permanent, auditable boundaries directly within a project repository. The Discovery Stage (Onboarding the Agent) Before an agent writes a single line of code, it must parse the overall project layout. These files act as the "map" for an incoming AI. llms.txt Serves as a lightweight text directory mapped out in Markdown format. Placed at the root of a project or webs

I learned this the hard way: when an LLM says “it matches the docs”, it can still be wrong for a boring reason—it didn’t read the part that matters. I’m building a small SaaS (checklists as a service). No users yet. Plenty of documentation already. And at some point my docs stopped being an asset and started turning into a liability. This is the story of how I rebuilt my documentation so that an LLM could actually read it end-to-end —and how that restructure helped me. The moment I got scared: “silent misses” The docset grew. I kept asking the LLM to verify tasks against it. And then I noticed a pattern that felt worse than hallucinations. Not “the model invented stuff”, but “the model confidently said it matches ”—while quietly missing exceptions, prohibitions, and thresholds. Keyword scanning instead of reading. I called it silent drift : code slowly moves away from conventions, while the invariants remain only in my head. In a project with roles, audit, and CI/CD security gates, that kind of drift isn’t “just messy docs”. It’s how you lose the ability to implement and review changes consistently. I couldn’t do it manually (and I couldn’t delegate it fully) I knew I had to redo the documentation. But I also knew I couldn’t realistically do it all by hand. At the same time, I couldn’t just tell an LLM: “Rewrite everything according to approach X.” Not enough context, too easy to lose control. So I went with a third option: build a reliable process out of unreliable components— me + an LLM . Step 1: I separated my docs into domains (and forced the model to actually read) First, I extracted domain areas from the old documentation—the vocabulary I was using to describe the project and its parts. I tried to keep domains mutually independent (so the overall framework stays holdable in my head). Then I ran the same loop for each domain: I asked the LLM to read all old docs carefully and extract requirements for that domain. I moved those requirements into a dedicated fil

A hands-on knowledge-distillation project: Qwen2-VL-7B → 2B for UI-screenshot understanding, trained, evaluated and benchmarked end-to-end on an M4 Pro. 2.4× faster — and why the teacher lost on ROUGE-L.

The dominant production failure mode for LLM agents in 2026 isn't bad reasoning — it's capacity. Here's what the data shows, why nobody demos it, and the capacity-engineering patterns that actually keep agents alive under load.

AI Builder Notes - May 2026 AI-assisted notes from my liked-tweets feed, organized around agent workflows, browser traces, model loops, and guardrails. Practical takeaways Start with the workflow, not the agent. A useful agent task has a source of truth, a narrow action, a verifier, and a stop condition. “Review this repo” is vague. “Find auth bugs in these routes, cite file lines, run the relevant tests, and stop after the first credible exploit path” is a workflow. Use dynamic workflows in claude code - to do the vibe bits for thinking through a workflow. Think of it like this - you can describe in natural language an entire workflow consisting of multiple agents at various steps - I want the docs updated, tests passed, security review done and also playwright tests done. Dynamic workflows figures out which parts can be divided in parallel and what should be done sequentially. Creates a flowchart - and writes JS code for it. Its a JS script that can execute subagents at scale and deterministically [1] Planner/executor split is the way to go. Spend the expensive model on taste, decomposition, and risk discovery. Use cheaper or narrower models for repeatable implementation once the task has tests, rubrics, logs, or examples. [2] Do not judge an agent workflow by the model name alone. If the loop has repo access, a rubric, a way to inspect tool calls, and a verifier, a less fashionable model can still do useful work. The Letta Code / GLM 5.1 review-bot example is interesting for that reason, not because “someone used X instead of Y” is interesting by itself. [3] Prefer small interfaces to giant tool menus. MCP tool call definitions are rotting your context! The monday.com GraphQL example was the clearest cost warning: one task used 15k tokens through SDK/code-mode and 158k tokens through a real MCP server. MCP is useful, but a menu of tools is not automatically an efficient interface. [4] [5] For browser work, save the trace. Run the workflow once, inspect wasted act

Large language models (LLMs) understand and generate text from prompts. OpenAI exposes models through the Responses API . The official openai npm package is the practical way to call it from Node.js. This post covers common patterns beyond a single prompt string. Prerequisites OpenAI account Generated API key Enabled billing Node.js version 26 openai package installed ( npm i openai ) For Markdown output: marked , dompurify , and jsdom ( npm i marked dompurify jsdom ) Client setup Create a client with your API key (read from the environment in production). import OpenAI from ' openai ' ; const client = new OpenAI ({ apiKey : process . env . OPENAI_API_KEY }); The same SDK can target other hosts that implement a compatible API by setting baseURL and apiKey : const client = new OpenAI ({ apiKey : process . env . LLM_API_KEY , baseURL : ' https://your-gateway.example/v1 ' , }); Azure OpenAI uses AzureOpenAI instead. Many third-party gateways support Chat Completions only; the examples below use client.responses.* , so confirm your provider supports the Responses API (especially for tools like web search). Basic integration Pass a string as input and read output_text from the response. const response = await client . responses . create ({ model : ' gpt-5.5 ' , input : ' Write a one-sentence bedtime story about a unicorn. ' , }); console . log ( response . output_text ); System prompt Use top-level instructions for stable behavior (tone, format, role). They take precedence over casual wording in the user message. const response = await client . responses . create ({ model : ' gpt-5.5 ' , instructions : ' Reply in one short sentence. Use plain language. ' , input : ' Explain what an LLM is. ' , }); console . log ( response . output_text ); Few-shot prompting Pass prior turns as an input array with user and assistant roles, then the new user message. Keep task rules in instructions . const response = await client . responses . create ({ model : ' gpt-5.5 ' , instructions :

When I decided to ship this blog in four languages, I had a clear mental ranking. English would win on volume. Spanish would be runner-up because of the sheer speaker count. Japanese would stay steady because it's my native language. Portuguese, I figured, was the long tail. I added it mostly out of completism. Twenty-two days later, the GA4 snapshot disagrees with every part of that ranking. PT: 748 pageviews , 709 sessions EN: 195 pageviews , 176 sessions JA: 27 pageviews , 29 sessions ES: 7 pageviews , 7 sessions That is Portuguese pulling roughly 3.8× English, 28× Japanese, and 107× Spanish on the same blog, same publishing cadence, same author. One Portuguese article on its own (a post about a 24-hour security agent: 375 PV) got more pageviews than my entire English blog combined. I wrote that article hoping Spanish would surprise me. Instead Portuguese surprised me, and Spanish quietly continued to not exist. The setup, so you can discount my numbers properly This is not a clean comparative experiment. It's a single blog, kenimoto.dev , running four language directories ( /en/ , /ja/ , /pt/ , /es/ ). Articles get translated through a cross-language LLM pipeline, then hand-edited for register and locale (BR Portuguese vs PT Portuguese, LatAm-neutral Spanish vs Spain Spanish). The window: 2026-04-30 to 2026-05-21, 22 daily snapshots. EN has 26 articles. JA has 25. PT has 17. ES has 10. So PT has fewer articles than EN and still beats it almost 4 to 1. If you stop reading here, take this one thing: language asymmetry can swallow article-count asymmetry whole . Adding articles in a saturated language is slower than adding articles in an underserved one. Why Portuguese pulled ahead I don't think the answer is "Portuguese readers like me more." I think three asymmetries are stacking on top of each other. 1. TabNews is a community door English doesn't have TabNews is a Brazilian developer community where you can post a technical article and have it actually read by h

Originally published on rikuq.com . Republished here for Dev.to's readers. The FinOps Foundation Framework is the reference architecture for cloud financial management. It's been maintained by the FinOps Foundation (a Linux Foundation project) since 2018 and has matured into the de facto standard most serious cloud cost work is built on. In 2026 it received a substantial refresh that extended its scope from pure cloud spend to include AI/ML, SaaS, licensing, and broader technology categories. For practitioners thinking about formalising FinOps practice — or evaluating providers who claim to do FinOps — knowing what the Framework actually covers is what separates a real implementation from a marketing label. This post walks through the Framework structure, the 2026 updates, and how it applies specifically to AI/ML spend. I'm Ravi. I run three production AI SaaS solo ( Prism , Citare , BatchWise ) and do advisory work on FinOps via rikuq services . The walkthrough below is what I use when teams ask "what does the FinOps Foundation Framework actually look like in practice?" TL;DR Element What it is Phases Three concurrent operational modes: Inform, Optimize, Operate Principles Six foundational principles guiding all FinOps practice Capabilities The functional areas of activity a FinOps practice covers Personas Engineering, Finance, Procurement, Leadership, Operations, ITAM, Sustainability 2026 additions Executive Strategy Alignment, Technology Categories taxonomy, Converging Disciplines recognition AI/ML extension New Technology Category with specifics on GPU/CPU differential, token pricing, make-vs-buy economics The six foundational principles Before the structural mechanics, the Framework's six principles establish the cultural and operational mindset. They're worth knowing because they're how the Framework's authors test whether something is "really" FinOps or just cloud cost cutting. Teams need to collaborate. Engineering, Finance, Procurement, and Business teams w

How to make LLMs deterministic, in plain English. The version I share with founders and product teams before they make decisions worth real money. You use AI tools every day. But can you explain what happens when you hit send? Most people cannot. And that gap is costing them. Bad prompts. Broken products. Decisions made on the wrong assumptions. The Hard Truth Every LLM explainer out there is written for researchers or so basic it tells you nothing useful. Neither helps you build better products or work with AI more effectively. This is the version I share with senior leaders, founders, and product teams before they make decisions worth real money. 1. It Is Not a Search Engine. It Is Not a Database. It Is a Prediction Machine. When you type a prompt and hit send, the LLM is not finding an answer from somewhere. It is predicting the most likely words to follow your input. Based on patterns it learned from billions of documents. That is the whole process. Wrong: "The AI knows the answer." Right: "The AI predicts the most likely answer based on what it has seen." This changes everything about how you use it. When an AI gives you a wrong answer confidently, it is not broken. It is doing exactly what it was built to do. Predict. Not verify. 2. The Autocomplete Comparison (And Why It Only Gets You Halfway) You have probably heard the phrase "autocomplete on steroids." It is not wrong. But it misses something important. Your phone autocomplete learned from your messages. An LLM learned from most of the written internet. Books. Research papers. Code. Billions of examples. At that scale, the patterns start to look a lot like real thinking. Not because the model understands in the way you do. Because it has seen so much that it can predict what a good answer looks like. When I was building AstroNayak I fed Vedic astrology principles into the system prompt. The LLM produced interpretations that genuinely surprised me. It did not know Vedic astrology. It had seen enough of it t

The row looked perfect. rating: 7 . Valid JSON, right type, no nulls, no missing keys. My schema check waved it through. The page had returned HTTP 200. The selectors hadn't moved. Everything green. A rating of 7 on a 5-star site is impossible. The model invented it, formatted it correctly, and handed it to me with total confidence. That's the failure I want to talk about. Not the scraper that breaks loudly. The one that hands you a clean-looking row that is quietly, plausibly false — and sails past every check you have, because your checks are all looking at the shape of the data, and the lie is in the value . TL;DR HTTP 200, intact selectors, and valid JSON tell you the form is fine. They say nothing about whether the value is true. When an LLM extracts from messy free-text, structured-output mode guarantees you get valid JSON. It does not guarantee the content is real. The model fills uncertain fields rather than leaving them empty — because the schema demands a complete row. A ~60-line value-level sanity gate (ranges, dates, cross-field, reference, language) catches the obvious lies before they hit your database. Real code and real output below. The honest catch: this gate catches rule violations , not plausible lies inside the allowed range . A rating: 4 where the truth is 2 slides right through. I'll be specific about where the gate stops. Two different ways a scraper lies to you I wrote about source drift last week — the case where the page changes underneath you and a 30-line schema check catches the structure shifting. That's an input problem. The source mutated; your agreement with the page broke; you detect it by watching the shape. This is the other end of the pipe. The source is fine. The page is intact, the selectors are correct, the structure is exactly what you expected. The thing that lied to you is the model , on the extraction step, when you asked it to pull structured fields out of a paragraph of human prose. Those two failures feel similar and t

In This Article The Deal Intelligence Gap Data Model Design Transcript Analysis with Claude Automated Follow-Up Drafting PostgreSQL JSONB Storage Putting It Together The Deal Intelligence Gap Most CRM systems are excellent at storing what happened — call logged, email sent, stage updated — and poor at capturing what was learned. A sales call produces qualitative intelligence that is genuinely valuable for deal strategy: what objections surfaced, how strongly the prospect signaled interest, what next steps were agreed to, and what risk flags the conversation revealed. That intelligence almost never makes it into the CRM because it requires someone to spend 15 minutes synthesizing unstructured notes into structured fields. Large language models change this equation. Given a call transcript, Claude can extract structured deal intelligence in seconds — categorizing sentiment, identifying specific objections, recommending stage movement, and flagging risk signals — with accuracy that equals or exceeds what a well-trained sales analyst would produce manually. Data Model Design The data model centers on two tables. The deals table stores core deal attributes as a JSONB column, which allows flexible schema evolution without migrations as the intelligence fields change over time. The deal_activities table records each interaction — calls, emails, meetings — with the raw content in TEXT and the extracted intelligence in a separate JSONB column. A GIN index on both JSONB columns enables fast attribute queries across the deal pipeline. CREATE TABLE deals ( id UUID PRIMARY KEY DEFAULT gen_random_uuid (), company TEXT NOT NULL , contact TEXT , stage TEXT , attributes JSONB DEFAULT '{}' , created_at TIMESTAMPTZ DEFAULT now (), updated_at TIMESTAMPTZ DEFAULT now () ); CREATE TABLE deal_activities ( id UUID PRIMARY KEY DEFAULT gen_random_uuid (), deal_id UUID REFERENCES deals ( id ), activity_type TEXT , raw_content TEXT , intelligence JSONB , created_at TIMESTAMPTZ DEFAULT now () )

About this series. I'm going to take a fresh paper - Self-Distilled Agentic Reinforcement Learning (SDAR, arXiv:2605.15155 ) - and architect it end to end on AWS: the system design, the actual gate code, the evaluation plan, and a brutally honest cost model. What I'm not going to do is wave a benchmark number around. Reproducing a paper like this costs thousands in GPU time, and I'd rather show you the machinery than a screenshot you can't audit. The design is the deliverable. This is Part 1. A small, infuriating problem Picture an LLM agent working a web-shopping task. It reads the goal, searches, clicks a category, filters, opens a product, compares, adds to cart - twelve steps in all. At the end, it bought the wrong thing. So you do what reinforcement learning tells you to do: you score the trajectory. Reward = 0. Bad agent. Now answer this: which of the twelve steps was actually wrong? Maybe step 3, the search query, was fine and step 9, a filter choice, doomed everything. Maybe steps 1–11 were brilliant and step 12 fat-fingered the wrong button. Your single scalar reward has no idea. It punishes all twelve equally, including the eight that were correct. That's the supervision problem in agentic RL, and it's the thing this whole series is about. Why "just use RL" isn't enough for agents RL has become the default way to post-train LLM agents. The catch is that the reward usually lands at the trajectory level - one number for the entire multi-step episode. For a single-turn task ("answer this question"), that's tolerable; the action and the outcome are close together. For a long-horizon agent - ten, twenty, fifty turns of searching, calling tools, and reacting to an environment - it's a disaster of credit assignment . The signal is too coarse to tell the model which decisions earned the reward and which torpedoed it. You can throw more episodes at it and let statistics sort the credit out eventually. But "eventually" on a 30-turn task burns a lot of expensive comp