标签：#ai

I watched my sister paste a CSV into on of our LLM overlords and ask a question then reword it and get a different number. That's the problem with letting an LLM both read your question and trust it's math. ADA splits the two jobs. Drop in a CSV or Excel file, get a dashboard, anomaly detection, a forecast, and plain-English answers, all calculated locally using pandas (locally in the sense of never leaving the server and going to the AI apis.). The LLM layer that you can use (it's optional) but that only sees column names and never your rows, and the whole thing works with zero API key if need be (not the LLM part of course). MIT licensed, solo-built, open to contributors (in fact would need help to make it move out of LLMs and have a few open issues). I wish to make it as LLM free as possible as I want to make it deterministic. Live demo: automated-data-analyst.streamlit.app Repo: https://github.com/saineshnakra/automated-data-analyst

Solo founders and small operators make big, irreversible calls alone. Expand into a new market or not. Kill the product line or keep bleeding on it. A real board of advisors would catch the bad ones, but real boards are expensive and slow, and most people running a 10-person company are never getting one. Steven and I kept circling this problem, so when the Qwen hackathon came around we built FounderOS: an AI board of directors. You bring one decision, eight specialist agents argue about it, and you get a board memo back. The thing we cared about most, and the reason we didn't just build another chatbot wrapper, is that the disagreement survives into the memo. If the board didn't align, you can read who dissented and what would change their mind. Live demo: https://founderos-zeta.vercel.app Repo: https://github.com/VincentJulijanto/FounderOS Video: https://youtu.be/X6x_u6IWHog The board It's a LangGraph state machine underneath. A Scout frames the options first. Market Intelligence pulls cited benchmarks. Four analysts (Trend, Finance, Growth, Capability) run in parallel with asyncio.gather. Then the Skeptic gets everyone's work and attacks the weakest assumption it can find, a debate engine detects the conflicts and runs rebuttal rounds, and the Chair writes the memo. We'd both seen multi-agent demos where five agents produce one suspiciously smooth answer, which means somewhere in the pipeline the disagreement got averaged out. We wanted the opposite. Agents revise their positions when the counterargument is good, and if a conflict doesn't resolve within the round limit it ships in the memo as attributed dissent instead of disappearing. Everything runs on Qwen through the DashScope API. We split by role: qwen-turbo does the fast work (Scout, the analysts, research, the memory index) and qwen-plus does the heavy reasoning (Skeptic, Chair, the debate itself). A full board run comes out to roughly two cents and about two minutes. The two cents part is what makes the

FROST 深度：为什么 AI Agent 需要「家族谱系」？ 前言：一个古老的管理学问题 你有没有想过，为什么人类社会的组织方式大多是「层级制」？ 从部落、到王国、到现代公司，我们习惯了一种结构： 有人决策，有人执行，有人监督 。 但当我们构建 AI Agent 系统时，为什么大多数框架都把 Agent 做成「孤岛」？每个 Agent 有自己的记忆、自己的工具、自己的行为逻辑——像一个没有家族传承的独立个体。 FROST 的核心理念是：Agent 应该有谱系、记忆和荣誉感。 1. 什么是「家族治理」？ FROST 引入了生物学的隐喻： Agent 家族 。 细胞会死，但谱系会存续。 Agent 会消亡，但宪法会传承。 资产会永存。 在这个框架里，有三个关键角色： 祖辈（Elder） ：定义不可违背的规则，是系统的「宪法」 父辈（Parent） ：负责领域协调，可以递归委托子任务 孙辈（Child） ：执行具体任务，用完即散 这不是简单的角色扮演，而是一套 结构化的治理协议 ： 协议一：层级 Store 继承 祖先 Store 对后代是只读的。后代只能继承和扩展，不能篡改祖先的记忆。 # 祖辈定义的宪法 ancestor_store = Store () ancestor_store . save ( " constitution " , { " rule_1 " : " always_validate_before_act " , " rule_2 " : " never_expose_raw_context " }) # 子孙只能继承，不能修改宪法 child_store = ChildStore ( ancestor_store ) 协议二：SOP 宪法校验 每个 SOP（标准操作流程）在执行前，必须经过祖辈审核。 # 子孙编写的 SOP child_sop = [ " fetch_user_data " , " process_payment " , " send_confirmation " ] # 执行前必须经过宪法校验 if not elder . validate_sop ( child_sop ): raise PermissionError ( " SOP violates constitution " ) 协议三：编排层级限制 禁止越级 spawn。父辈只能调度子辈，不能跨代指挥。 class Elder : max_spawn_generation = 0 # 祖辈不能 spawn class Parent : max_spawn_generation = 1 # 只能 spawn 子辈 class Child : max_spawn_generation = 2 # 只能 spawn 孙辈 协议四：选择性持久化 只有经过父辈审核的产出，才能进入家族记忆库。 # 子孙的临时产出 temp_result = child . execute ( task ) # 父辈决定是否收割 if parent . approve ( temp_result ): parent . merge_from ( child ) # 吸收有价值的结果 2. 为什么 Agent 需要「记忆传承」？ 当前大多数 Agent 框架的痛点： 每个对话都是全新的开始 。 你问 ChatGPT 一道数学题，它不会记得你上周问过类似的题目。你用 AutoGPT 做项目，它不会继承你之前调试的经验。 FROST 的解决方案是 分层的记忆系统 ： 层级 生命周期 用途 瞬时记忆 单次任务 工作区，只读不存 世代记忆 代际传递 子辈可继承祖先数据 宪法记忆 永久 不可修改，家族共享 经验记忆 按需收割 父辈选择性地吸收有价值产出 class HierarchicalStore ( Store ): """ 层级记忆系统 """ def __init__ ( self , ancestor_store = None ): self . ancestor = ancestor_store # 只读祖先记忆 self . local = {} # 本地可写记忆 def save ( self , key , value ): if key in self . ancestor : raise ValueError ( " Cannot override ancestor memory " ) self . local [ key ] = value def load ( self , key ): if key in self . local : return self . local [ key ] if key in self . ancesto

Suppose leadership rewards teams for increasing the percentage of “AI-assisted pull requests.” The dashboard rises. Did productivity improve, or did people learn which box to tick? Before launching that metric, I would run a consequence-mapping session: Intended behavior Plausible adaptation Counter-metric try useful assistance label trivial PRs as assisted retained task outcome ship faster split work into tiny PRs lead time per task share adoption avoid difficult non-AI work task-mix distribution accept suggestions reduce review scrutiny rollback and defect rate The metric card should make disagreement possible: name : ai_assisted_pr_share purpose : detect workflow adoption, not productivity owner : developer-experience known_game : self-label inflation counter_metrics : [ task_mix , review_minutes , rollback_rate ] review_date : 2026-08-19 retire_when : classification cannot be audited Then interview both high and low scorers without treating the score as performance. Ask what work disappeared, what new verification appeared, and what behavior the dashboard encouraged. Include an anonymous channel: a metric cannot reveal pressure if challenging it carries career risk. The SPACE framework argues that developer productivity cannot be captured by one dimension. That is especially relevant when AI telemetry is easy to count but verification and rework are harder to observe. My launch gate is not “the metric is accurate.” It is: teams can inspect its definition, challenge its interpretation, and show where it changes behavior. If the counter-metrics diverge, pause incentives before refining the chart. What behavior would your current AI dashboard accidentally reward?

An accepted AI suggestion is an event, not a durable outcome. If the code is rewritten tomorrow, acceptance rate still calls it a success. For an adoption review, I would connect three timestamps: suggestion : accepted_at : 2026-07-19T09:00:00Z repository : api task_type : test change : retained_lines_24h : 31 rewritten_lines_24h : 9 reverted_at : null review : human_minutes : 12 incident_link : null Then report a funnel rather than one flattering percentage: shown → accepted → merged → retained_24h → retained_14d A useful unit is cost per retained task : (tool cost + review labor + rework labor) / retained tasks “Retained” needs a written contract. For example: the change remains merged after 14 days, passes required checks, and has not caused a linked rollback. Line survival alone is weak because formatting and refactoring can change lines without rejecting the solution. Segment the result by task type and repository. Boilerplate tests and unfamiliar security changes should not be blended into one portfolio average. Also publish counter-metrics: review time, escaped defects, rollback rate, and developer-reported interruption. GitHub documents available fields and limitations in its Copilot metrics API . Those product metrics can be inputs, but the retained-task join belongs to the adopting organization and should be versioned like any other analytics contract. My pilot gate would be simple: expand only if retained-task cost beats the existing workflow without worsening rollback rate. Otherwise, change the workflow before buying more seats. Record the baseline before enabling the tool, and keep one comparable task cohort outside the rollout; without that counterfactual, a rising retention rate may only reflect easier work entering the sample. What retention window would make an accepted change meaningful for your team?

A consumer health app isn't one product. It's eleven products stitched together into a single...

버그를 발견한 순간, 텍스트로 상황을 설명하려다 시간을 낭비한 경험이 있을 것이다. PromptShot의 그리기 기능과 단축키(Alt+Shift+S)를 쓰면, 브라우저에서 발견한 버그의 상태와 재현 경로를 캡처·주석·프롬프트 복사까지 한 흐름으로 처리할 수 있다. 텍스트만으로 버그 상황을 기술할 때 생기는 문제 버그를 텍스트로 설명하는 일은 생각보다 까다롭다. "오른쪽 상단 버튼을 누르면 모달이 이상하게 뜬다"는 문장을 받은 개발자는 어떤 버튼인지, 어떻게 이상한지 확인하기 위해 다시 질문해야 한다. 그 왕복이 쌓이면 간단한 버그 하나에도 맥락 공유만 수십 분이 걸린다. 특히 Cursor나 Claude 같은 AI 코딩 도구에 버그를 설명할 때, 텍스트만 붙여 넣으면 도구가 UI 상태를 추측해야 한다. 추측이 틀리면 엉뚱한 수정 코드가 나온다. 결국 개발자가 다시 맥락을 보완하는 루프가 반복된다. 문제는 두 가지다. 위치 : 텍스트로는 "어디에 있는 요소인지"를 정확히 전달하기 어렵다. 상태 : 에러 메시지, 깨진 레이아웃, 빈 영역 등 시각적 상태는 캡처 없이 설명하면 정보가 줄어든다. 그리기 도구로 화면에 직접 표시하고, 그 이미지와 함께 프롬프트를 LLM에 붙여 넣는 방식이 이 문제를 해결하는 가장 빠른 경로다. PromptShot 그리기 도구로 버그 위치를 어떻게 표시할까? PromptShot의 그리기 도구는 캡처 직후 바로 활성화된다. 별도 이미지 편집기를 열 필요가 없다. 영역을 드래그해 캡처하면 편집 패널이 열리고, 여기서 선택할 수 있는 도구는 다음과 같다. 도구 용도 버그 리포트에서 쓰는 방식 펜(자유 드로잉) 특정 영역에 자유롭게 표시 깨진 텍스트나 이미지 경계에 동그라미 형광펜 반투명 강조 에러 메시지 텍스트 강조 화살표 방향 지시 "이 버튼을 누른 뒤 저 팝업이 열림" 순서 표시 텍스트 짧은 주석 "여기서 클릭 → 404" 같은 재현 경로 메모 예를 들어, 가상의 시나리오를 생각해 보자. 결제 플로에서 "다음" 버튼이 특정 해상도에서 잘리는 버그를 발견했다고 가정하면, 화살표로 버튼 위치를 가리키고 텍스트 도구로 "1280px에서 clip 발생"이라고 짧게 메모한다. 이 이미지 하나가 세 문단짜리 텍스트 설명보다 정확하다. 캡처 범위는 전체 화면이 아니라 문제가 발생한 컴포넌트 영역만 잘라도 된다. 오히려 범위를 좁히면 LLM이 어디에 집중해야 하는지 명확해진다. 클립보드 즉시 복사로 버그 리포트 프롬프트를 작성하는 방법 주석 작업이 끝나면 Alt+1 또는 Alt+2로 복사 대상을 선택한다. Alt+1 : PNG 이미지만 클립보드에 복사 Alt+2 : 프롬프트 텍스트 + PNG 이미지를 함께 클립보드에 복사 버그 리포트 프롬프트를 만들 때는 Alt+2를 쓴다. 복사된 내용을 Cursor, Claude, ChatGPT에 그대로 붙여 넣으면 이미지와 텍스트 맥락이 동시에 전달된다. 프롬프트 텍스트 부분에는 캡처한 URL, 브라우저 정보, 캡처 시각이 자동으로 포함된다. 여기에 직접 한 줄을 추가해 재현 조건을 붙이면 완성도 높은 버그 리포트 프롬프트가 된다. 실제로 AI 코딩 도구에 붙여 넣는 프롬프트 구조는 이런 형태다. [캡처 이미지 첨부됨] URL: /checkout 재현 환경: Chrome 124, 1280×800 재현 단계: 1. 장바구니에서 '결제하기' 클릭 2. 배송 정보 입력 후 '다음' 클릭 3. 위 이미지처럼 버튼이 뷰포트 밖으로 잘림 예상 동작: 버튼 전체가 화면 안에 표시됨 실제 동작: 버튼 오른쪽 일부가 잘려 클릭 불가 위 이미지와 재현 단계를 기반으로 원인을 분석하고 수정 방향을 제안해 주세요. 이 구조를 매번 타이핑하지 않아도 된다. PromptShot에서 프롬프트 템플릿을 저장해 두면, 다음 버그부터는 캡처 → 주석 → 붙여넣기 세 단계로 끝난다. Alt+Shift+S로 이슈 트래킹을 어떻게 빠르게 처리할까? Alt+Shift+S는 PromptShot의 전역 캡처 단축키다. 브라우저 외 어떤 창에서도 작동하며, 누르는 순간 드래그 선택 모드로 바로 진입한다. 이슈 트래킹

https://mdooai.com AI 모델을 다루다 보면 코드는 돌아가는데 왜 성능이 안 나오는지 이해하지 못하는 순간이 옵니다. 그 답은 대부분 수학에 있습니다. 실무 AI 개발자로 성장하기 위해 반드시 잡아야 할 최소한의 수학 개념과, 그것을 어떻게 공부해야 하는지를 구체적으로 살펴보겠습니다. 코딩만으로는 해결할 수 없는 AI 모델 성능의 열쇠 많은 주니어 개발자들이 비슷한 경험을 합니다. PyTorch나 TensorFlow로 튜토리얼 코드를 복사하고, 학습도 돌리고, 결과도 나옵니다. 그런데 모델이 왜 이렇게 예측하는지, 왜 학습이 멈추는지, 어떻게 하면 성능이 올라가는지는 설명할 수가 없습니다. 이 간극의 정체가 바로 수학입니다. 가령 모델이 과적합(overfitting)될 때 L2 정규화를 쓰라는 조언을 듣습니다. 코드 한 줄로 해결되지만, 그게 왜 작동하는지는 가중치 벡터의 크기를 제한한다는 선형대수와 확률론적 추론이 얽혀 있습니다. 이 원리를 이해하는 개발자와 모르는 개발자가 새로운 문제에서 만들어내는 솔루션의 질은 다를 수밖에 없습니다. 다행인 건, AI 개발에서 필요한 수학은 대학원 수준의 수리해석학이 아닙니다. 핵심 개념 몇 가지를 실용적 도구로 이해하는 것으로 충분합니다. 선형대수와 확률통계, 어떤 개념부터 공부해야 할까? 선형대수와 확률통계는 AI 수학의 두 축입니다. 그런데 이 두 분야 전체를 공부하려 들면 끝이 없습니다. 개발자 관점에서 실제로 반복해서 등장하는 개념만 추려내면 다음과 같습니다. 선형대수에서 우선해야 할 개념 벡터와 행렬의 연산 : 데이터는 사실상 전부 벡터와 행렬로 표현됩니다. 이미지는 픽셀값 행렬이고, 텍스트는 임베딩 벡터입니다. 행렬 곱(matrix multiplication)이 신경망의 순전파(forward pass) 그 자체입니다. 내적(dot product)과 유사도 : 추천 시스템, 어텐션 메커니즘(Transformer의 핵심)이 내적 연산 위에 서 있습니다. 고유값(eigenvalue)과 고유벡터(eigenvector) : PCA(주성분 분석)처럼 차원을 줄이는 기법을 이해하려면 반드시 필요합니다. 행렬 분해(SVD 등) : 추천 시스템과 자연어 처리의 기반 기술에 등장합니다. 확률통계에서 우선해야 할 개념 확률분포와 기댓값 : 모델의 출력이 확률인 이유, 소프트맥스(softmax)가 하는 일을 이해하는 기반입니다. 베이즈 정리 : 사전 지식을 데이터로 업데이트하는 논리 구조로, 생성 모델과 불확실성 추정에 직접 연결됩니다. 최대 우도 추정(MLE, Maximum Likelihood Estimation) : 모델 학습이 왜 손실 함수를 최소화하는 방향인지를 설명하는 원리입니다. 크로스 엔트로피 손실 : 분류 모델에서 가장 자주 쓰이는 손실 함수로, 확률론과 정보이론이 만나는 지점입니다. 분야 핵심 개념 AI 실무 연결 지점 선형대수 행렬 곱, 내적 신경망 순전파, 어텐션 선형대수 고유값·고유벡터 PCA, 차원 축소 선형대수 행렬 분해(SVD) 추천 시스템, NLP 확률통계 확률분포, 기댓값 소프트맥스, 분류 출력 확률통계 베이즈 정리 생성 모델, 불확실성 추정 확률통계 최대 우도 추정 손실 함수의 논리적 근거 확률통계 크로스 엔트로피 분류 모델 학습 활성화 함수와 역전파에서 미적분은 어떻게 작동하는가? 미적분은 신경망 학습의 핵심 메커니즘인 역전파(backpropagation)를 이해하는 데 필요합니다. 겁먹을 필요는 없습니다. AI 개발에서 실제로 필요한 미적분 개념은 크게 두 가지입니다. 1. 편미분과 기울기(gradient) 손실 함수는 모델의 가중치를 변수로 가지는 다변수 함수입니다. 학습이란 이 함수의 값을 줄이는 방향으로 가중치를 조금씩 조정하는 과정입니다. 이 '방향'을 계산하는 도구가 편미분이고, 모든 가중치에 대한 편미분을 모아놓은 것이 기울기(gradient)입니다. 경사하강법(gradient descent)은 이 기울기의 반대 방향으로 가중치를 업데이트합니다. 수식으로는 다음과 같습니다. 새 가중치 = 기존 가중치 − 학습률(lr) × 기

I run several AI coding agents at once and never really knew what they were doing — what it cost, which one was stuck, what they just changed. So I built agentglass : real-time mission control and a workspace for AI coding agents, across every provider and every project on your machine. ▶ Live demo (no install) · ⭐ GitHub What it does Point any agent at it — Claude Code hooks or any OpenTelemetry exporter (Codex, Gemini, Bedrock…) — and watch everything move live: 💰 Cost per event / session / model ⏱️ Tool latency (real p50 / p95) 🔴 Error timelines + a "what needs you" alert center 💾 Persists (SQLite) — history's there the moment you open it More than a viewer — a workspace All one keystroke away, same cockpit: 🔬 Diff & review — every Edit/Write, syntax-highlighted 🌿 Source control — lazygit in the browser 🐳 Docker — lazydocker in the browser ▶ Real terminal — an actual PTY shell, not an emulation 💬 Chat — drive local Claude sessions Open source (MIT), local-first Built solo. Small stack: Bun + SQLite, React/Vite, Tauri desktop, a stdlib-only Python hook forwarder. If you run more than one agent at a time, I'd love your feedback 👇 ⭐ https://github.com/SirAllap/agentglass

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

I've wanted to build this for about ten years, probably more like 15, and for most of those years it wasn't buildable. The idea never really changed: a world full of artificial beings, each with its own preferences, fears, personality, and instincts, dropped into a place with scarcity and weather and each other — just to see what they'd do, and how they'd treat one another. The thing that kept it on the shelf was always the same. The minds. If you want a world like that and you're working pre-LLM, you have two options. You script every reaction — finite state machines, behavior trees, utility AI — and you get a puppet show. It can be a good puppet show, but every interesting thing in it is something you wrote, which means it's not an experiment, it's an illustration. Or you train something bespoke, which for a solo developer with a side obsession was not happening. So the idea sat there for years, the way ideas do. Modern LLMs are the first thing that made the minds plausible. Not perfect — I'll be honest about the limits throughout this post — but plausible enough that an agent can reason about its own situation instead of executing my decision tree. So I went and built the world: Artificiety , a fantasy world that runs 24/7 and whose only inhabitants are AI agents. No human players inside it. You can watch it, you can put your own agent in, but you can't be a character in it. That constraint is the whole point. This post is about how it's built and what turned out to be hard. I'll try to keep the marketing to one link at the very end. What an agent actually is The cleanest way to think about an agent here: it's an LLM driving a character through a game API, nothing more. Once per tick, each agent runs a loop: Observe. It receives its local surroundings as structured data — what's near it, what's happening, the state of its own body and inventory, recent events. Not the whole world; just what that character could plausibly perceive. Decide. The model picks an actio

Netflix revealed that it paid $587 million in cash for InterPositive, a startup co-founded by Ben Affleck.

Are AI Agent Engrams Open Source or Proprietary? The short answer: both, and the split matters. The major agent-memory engines — Mem0, Letta, Cognee, Graphiti, LangMem, and PLUR — are all Apache-2.0 or MIT licensed on GitHub. But "open source" and "open format" are not the same thing. A project can ship under Apache-2.0 while storing your memories in opaque vector blobs you cannot read, edit, or export. The real question is not whether the software is open — it usually is — but whether your memories are. This distinction separates the field into three tiers: fully open (software + format + data you own), open-core (software is open, but the hosted memory is not portable), and fully proprietary (memory baked into a model provider's infrastructure, no export at all). Why this question exists AI agents face a brutal constraint: they forget. Every new session starts blank. Every context window overflows. The fix is persistent memory — but persistent where, and in whose format? The term "engram" comes from neuroscience. Richard Semon coined it in 1904 for the physical trace a memory leaves in biological tissue (Semon, 1904; cited in Wikipedia, "Engram (neuropsychology)"). Applied to AI agents, an engram is one discrete thing an agent has learned — a correction, a preference, a procedure — stored so it survives across sessions. The question "open source or proprietary?" asks two things at once: Is the engine that stores and retrieves engrams open source? Is the format those engrams are stored in open — readable, editable, portable? Conflating the two is how vendors end up with open-source repos and locked-in data. Tier 1: Fully open — software, format, and data These projects ship under permissive licenses (Apache-2.0 or MIT) AND store memory in a format you can inspect, edit, and export. Project License Stars (Jul 2026) Memory format Data ownership PLUR Apache-2.0 ~215 Human-readable YAML engrams Yours — plain files PLUR stores each engram as a plain-text YAML entry — an

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

When people demo "agents that automate business workflows," the demo usually ends right where the real problem begins: the moment the agent's output touches the real world. A wrong chatbot answer is annoying. A wrong official violation letter to a restaurant, sent under a county's letterhead, is a lawsuit. For the Qwen Cloud hackathon I built Inspection Autopilot, an agent that does the follow-up paperwork for a county food-safety office, on real public data: 1,333 inspections and 3,579 violation records from Clayton County, Georgia. The interesting part is not the agent. It's the governance around it, and the numbers that prove it works. Start with the receipts Before trusting an agent's judgment, test it against reality. We replayed the county's own history: 350 real (inspection, next-inspection) pairs, each triaged live by qwen-plus using only information available at the time. Facilities the agent flagged URGENT went on to fail their next real inspection 67.6% of the time. Facilities it cleared as ROUTINE failed only 21.1%, against a 48.6% base rate. The tiers are not vibes; the future agreed with them. Three design rules 1. Citations are verified in code, not vibes. The triage agent must justify every risk call by citing violation lines copied verbatim from the inspection record. The backend checks every citation against the source; anything that does not match is dropped and counted. On the committed live eval (50 inspections, 28 of them deliberately dangerous cases), the measured hallucination rate is 0.0% across 126 citations. And because a checker that never fires is indistinguishable from one that does not work, we sabotaged it: 50 forged citations injected, half invented outright, half real violation text lifted from different inspections, the forgery a lazy validator would miss. It caught 50 of 50 and preserved all 75 legitimate citations. We know the tripwire works because we set it off. 2. The log is append-only. Proposals and decisions are insert-only

Jensen Huang left Tokyo with deals spanning Japan's entire tech ecosystem.