Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Я заглянул туда. Там дохуя документации. Дикий геморой мусорки. Бесконечный склад, который вгоняет меня в панику. Но как инструмент это незаменимая часть Web. Я беру нужный мне чертёж и строю то, что мне нужно. window глобальный объект. Подключение к API старого браузера. Это Мозг, который даёт мне инструменты: Скелет HTML: (document) Память Хранилище: (localStorage) Сеть API: подключение к контрактам других серверов для сбора информации (fetch) Но главное, что я заценил это обработчик событий onload. Это и есть чудо архитектуры. Связь CSS, JS, HTML в корневой папке предка HTML. Я скидываю в него свой модуль, и он гарантирует, что всё запустится, когда скелет будет готов.

I've wanted to build this for about ten years, probably more like 15, and for most of those years it wasn't buildable. The idea never really changed: a world full of artificial beings, each with its own preferences, fears, personality, and instincts, dropped into a place with scarcity and weather and each other — just to see what they'd do, and how they'd treat one another. The thing that kept it on the shelf was always the same. The minds. If you want a world like that and you're working pre-LLM, you have two options. You script every reaction — finite state machines, behavior trees, utility AI — and you get a puppet show. It can be a good puppet show, but every interesting thing in it is something you wrote, which means it's not an experiment, it's an illustration. Or you train something bespoke, which for a solo developer with a side obsession was not happening. So the idea sat there for years, the way ideas do. Modern LLMs are the first thing that made the minds plausible. Not perfect — I'll be honest about the limits throughout this post — but plausible enough that an agent can reason about its own situation instead of executing my decision tree. So I went and built the world: Artificiety , a fantasy world that runs 24/7 and whose only inhabitants are AI agents. No human players inside it. You can watch it, you can put your own agent in, but you can't be a character in it. That constraint is the whole point. This post is about how it's built and what turned out to be hard. I'll try to keep the marketing to one link at the very end. What an agent actually is The cleanest way to think about an agent here: it's an LLM driving a character through a game API, nothing more. Once per tick, each agent runs a loop: Observe. It receives its local surroundings as structured data — what's near it, what's happening, the state of its own body and inventory, recent events. Not the whole world; just what that character could plausibly perceive. Decide. The model picks an actio

Я разбирал проект Lea Verou. 16 лет назад она создала костыль rgba.php, чтобы старые браузеры, не понимающие CSS-цвета (работающие на графическом дизайне в формате PNG-картинок), могли их отображать через слой старого браузера и новых технологий. Этот слой накладывался на старый браузер по PHP-семантической безликой связи API. Проблему не могут решить до сих пор. Весь Веб работает на этих костылях. Из-за них и появляются всё новые иллюзорные решения, которые с подавляющей сложностью становятся легаси. Даже решение Lea Verou со временем станет легаси для новых проблем бизнес-процессов. Старый браузер не исправить никогда. Если мы поменяем в нём всё то большая часть интернета просто обнулится. Банки, больницы и правительства, пока они на устаревших технологиях, будет вбухана ещё куча денег на создание костылей. Пришло понимание, что не существует идеальной архитектуры и правильного кода. Есть только ответственность в решении поставленных задач. То, что было написано в старых браузерных версиях, на которых мы находимся, останется с нами навсегда. Легаси не умирает. Оно просто покрывается новыми слоями костылей-решений. Инкапсуляция слоёв это не про чистый код. Это про то, как изолировать этот вечный хаос, чтобы наши проекты не перестали работать от чужого прошлого. Создание костылей — золотое время гениев. Архитекторы не строят идеальные замки. Они строят крепости, способные выдержать время.

Many of my favorite bloggers have stopped writing (or have drastically shifted topics). What are your favorite blogs? I miss the 2009 era of blogging about personal projects or super niche factoids.

AI agents that run order-to-cash operations Discussion | Link

Unify data analysis and plotting in one powerful workspace Discussion | Link

A native macOS terminal you can skin and theme Discussion | Link

The $35 camera comes in seven colors and has a slide cover.

"On the first attempt, reaching orbit, I never thought it was possible."

GitHub热门项目 | 《深入理解 AI Agent：设计原理与工程实践》（李博杰 著）开源主仓库：全书正文、编译版 PDF 与按章配套代码 | Stars: 5,299 | 1,734 stars today | 语言: Python

Last week I was reviewing the staking engine of a protocol before its mainnet launch. Deep in a 1,400-line contract, I found what looked like a serious bug. The reward math multiplied three values before dividing: uint256 delta = (lot.amount * rBase * midpointRate) / (RAY * RAY); Multiply-before-divide. If that intermediate product overflows uint256 , the whole epoch settlement reverts — and since every stake , withdraw , and setStake runs it, the post's funds get permanently frozen . That's a High-severity, fund-locking DoS. I had the finding half-written. lot.amount can reach 10M tokens ( 1e25 ). rBase grows with elapsed time. midpointRate can hit RAY . Multiply those and you blow past 2^256 ... I was ready to send it. Then I did the one thing that separates a real audit from false-positive spam: I checked the actual constants before writing the claim. uint256 private constant RAY = 1e18; // I'd assumed 1e27 RAY was 1e18 , not the 1e27 I'd been carrying in my head. And the interest rate had a hard cap — MAX_RATE_MAX_RAY = 5e18 , enforced even against a fully-captured timelock. I ran the numbers with the real values: For that multiplication to overflow, the protocol would need to go 2.3 × 10¹⁵ years without a single state update. Not reachable. The bug didn't exist. I deleted the finding. Why this matters more than the bug would have If I'd sent that report, here's what happens: the team's engineer clones the repo, plugs in the real constants, and realizes in ten minutes that I flagged an overflow that can't happen. Every other finding in my report now gets read with a raised eyebrow. My credibility — the entire product — is gone. This is the dirty secret of automated smart-contract auditing: the bottleneck isn't finding issues. It's not drowning the real ones in false positives. Anyone can run a scanner and paste 40 "criticals." A team that has to triage 40 flags to find the 2 that matter will — correctly — stop trusting you. The bar I hold: zero false positives o