标签：#ai

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

I've wanted to build this for about ten years, probably more like 15, and for most of those years it wasn't buildable. The idea never really changed: a world full of artificial beings, each with its own preferences, fears, personality, and instincts, dropped into a place with scarcity and weather and each other — just to see what they'd do, and how they'd treat one another. The thing that kept it on the shelf was always the same. The minds. If you want a world like that and you're working pre-LLM, you have two options. You script every reaction — finite state machines, behavior trees, utility AI — and you get a puppet show. It can be a good puppet show, but every interesting thing in it is something you wrote, which means it's not an experiment, it's an illustration. Or you train something bespoke, which for a solo developer with a side obsession was not happening. So the idea sat there for years, the way ideas do. Modern LLMs are the first thing that made the minds plausible. Not perfect — I'll be honest about the limits throughout this post — but plausible enough that an agent can reason about its own situation instead of executing my decision tree. So I went and built the world: Artificiety , a fantasy world that runs 24/7 and whose only inhabitants are AI agents. No human players inside it. You can watch it, you can put your own agent in, but you can't be a character in it. That constraint is the whole point. This post is about how it's built and what turned out to be hard. I'll try to keep the marketing to one link at the very end. What an agent actually is The cleanest way to think about an agent here: it's an LLM driving a character through a game API, nothing more. Once per tick, each agent runs a loop: Observe. It receives its local surroundings as structured data — what's near it, what's happening, the state of its own body and inventory, recent events. Not the whole world; just what that character could plausibly perceive. Decide. The model picks an actio

Netflix revealed that it paid $587 million in cash for InterPositive, a startup co-founded by Ben Affleck.

Are AI Agent Engrams Open Source or Proprietary? The short answer: both, and the split matters. The major agent-memory engines — Mem0, Letta, Cognee, Graphiti, LangMem, and PLUR — are all Apache-2.0 or MIT licensed on GitHub. But "open source" and "open format" are not the same thing. A project can ship under Apache-2.0 while storing your memories in opaque vector blobs you cannot read, edit, or export. The real question is not whether the software is open — it usually is — but whether your memories are. This distinction separates the field into three tiers: fully open (software + format + data you own), open-core (software is open, but the hosted memory is not portable), and fully proprietary (memory baked into a model provider's infrastructure, no export at all). Why this question exists AI agents face a brutal constraint: they forget. Every new session starts blank. Every context window overflows. The fix is persistent memory — but persistent where, and in whose format? The term "engram" comes from neuroscience. Richard Semon coined it in 1904 for the physical trace a memory leaves in biological tissue (Semon, 1904; cited in Wikipedia, "Engram (neuropsychology)"). Applied to AI agents, an engram is one discrete thing an agent has learned — a correction, a preference, a procedure — stored so it survives across sessions. The question "open source or proprietary?" asks two things at once: Is the engine that stores and retrieves engrams open source? Is the format those engrams are stored in open — readable, editable, portable? Conflating the two is how vendors end up with open-source repos and locked-in data. Tier 1: Fully open — software, format, and data These projects ship under permissive licenses (Apache-2.0 or MIT) AND store memory in a format you can inspect, edit, and export. Project License Stars (Jul 2026) Memory format Data ownership PLUR Apache-2.0 ~215 Human-readable YAML engrams Yours — plain files PLUR stores each engram as a plain-text YAML entry — an

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

When people demo "agents that automate business workflows," the demo usually ends right where the real problem begins: the moment the agent's output touches the real world. A wrong chatbot answer is annoying. A wrong official violation letter to a restaurant, sent under a county's letterhead, is a lawsuit. For the Qwen Cloud hackathon I built Inspection Autopilot, an agent that does the follow-up paperwork for a county food-safety office, on real public data: 1,333 inspections and 3,579 violation records from Clayton County, Georgia. The interesting part is not the agent. It's the governance around it, and the numbers that prove it works. Start with the receipts Before trusting an agent's judgment, test it against reality. We replayed the county's own history: 350 real (inspection, next-inspection) pairs, each triaged live by qwen-plus using only information available at the time. Facilities the agent flagged URGENT went on to fail their next real inspection 67.6% of the time. Facilities it cleared as ROUTINE failed only 21.1%, against a 48.6% base rate. The tiers are not vibes; the future agreed with them. Three design rules 1. Citations are verified in code, not vibes. The triage agent must justify every risk call by citing violation lines copied verbatim from the inspection record. The backend checks every citation against the source; anything that does not match is dropped and counted. On the committed live eval (50 inspections, 28 of them deliberately dangerous cases), the measured hallucination rate is 0.0% across 126 citations. And because a checker that never fires is indistinguishable from one that does not work, we sabotaged it: 50 forged citations injected, half invented outright, half real violation text lifted from different inspections, the forgery a lazy validator would miss. It caught 50 of 50 and preserved all 75 legitimate citations. We know the tripwire works because we set it off. 2. The log is append-only. Proposals and decisions are insert-only

Jensen Huang left Tokyo with deals spanning Japan's entire tech ecosystem.

Just why? ʅ͡͡͡͡͡͡͡͡͡͡͡(̸̢̛̼̞̭͋ͅ)̸͚̰͛̔̾̀̿͒͂:̴͓̞̑̌̂̆̊͋̀:̸͎̟̯̂̓̌ ҉ ͡ ͞ ͞ ͞ ҉● ࿀ ● ࿀ ● ҉⃝ ⃝͢ ͞ ͘ ͞⃝̕ ͢ ̛ ⃝ ̸ ̡ ͢⃝̧ ͡ ͡ ̀ ̧ ̢⃝͜ ҉ ͞ ͞ ⃝͞ ͞ ͡⃝ ⃝҉҈҉҈҉҈҉҈҉҈҉ :̶̢͙͙͕̠̩͆(̷̮͍͚̫͚͂̍)̵̳̗̊( ̟̞̝̜̙̘̗̖҉̵̴̨̧̢̡̼̻̺̹̳̲̱̰̯̮̭̬̫̪̩̦̥ What am I supposed to do with this? It's un-Googleable. Unpronounceable. It doesn't even render the same way on various platforms. It looks one way on YouTube Music. Another on Apple Music. And yet another on Bandcamp. Even the artist name ⣎⡇ꉺლ༽இ•̛)ྀ◞ […]

GenPage is a generative AI system developed by Netflix to replace its traditional multi-stage recommendation pipeline by directly generating personalized user homepages. GenPage leverages user history and request context as a prompt to produce the entire page, resulting in improved user engagement and reduced serving latency. By Sergio De Simone

On the latest episode of Equity, we debate whether Apple's lawsuit will cast over OpenAi's much-discussed plans to get into hardware and go public.

My Zero to Blockchain Engineer journey — Week 2 update Last week I announced I was teaching myself blockchain engineering from zero. This week I went deep into Cyfrin Updraft — and I have to be honest, it changed how I think about learning Web3. Here's everything I covered, what surprised me, and why I think this platform is the best free resource for anyone serious about becoming a blockchain developer. What I covered this week I'm currently 45% through the Blockchain Basics course on Cyfrin Updraft. Here's exactly what the lessons covered: Section 1 — What Is A Blockchain? This section answered the question properly — not in a surface-level way, but by starting with the problem blockchain solves. The lesson on centralized control hit differently. Think about it in a Nigerian context — banks freezing accounts, payment processors blocking transactions, elections where results can't be independently verified. These aren't hypothetical problems. They're things we live with. Blockchain's answer: a shared ledger where truth is verified by math, not by a middleman. That framing made everything click for me. Key lessons I completed: What Is A Blockchain — centralized problems vs decentralized solutions History Of Blockchain — from the double-spend problem to Bitcoin to Ethereum Benefits Of Blockchain — permissionless, immutable, credibly neutral Use Cases — DeFi, cross-border payments, DAOs, digital ownership Many Many Chains — L1s, L2s, Mainnets vs Testnets The Oracle Problem — why smart contracts can't access real-world data alone The Purpose Of Smart Contracts — math-based promises vs trust-based agreements What Is The EVM — Ethereum's computation engine explained simply Benefits Of Smart Contracts — decentralization, immutability, transparency Section 2 — Sending Transactions This is where things got hands-on — and where Cyfrin pulled ahead of every other platform I've tried. Key lessons: What Is A Wallet — your public address as a digital mailbox Setting Up A Wallet

The security officer said no. Not "maybe with some modifications." Not "let us review the data handling policy." No. Cloud-hosted AI code review was not going to happen in their environment. The codebase contained classified material. The network was air-gapped. No data leaves the building. End of conversation. This was a defence contractor. Their development team was writing code that couldn't touch any external API, any cloud service, any SaaS platform. But they were also writing 200,000+ lines of code per year with a team that was struggling to maintain review quality across twelve engineers, the exact problem that AI code review solves. The challenge: deploy AI code review capability that runs entirely on-premise, in an air-gapped network, with no external connectivity, while meeting the security and compliance requirements of a classified computing environment. This is the guide we wish existed when we started that project. It covers hardware requirements, model selection, inference server setup, CI/CD integration and the security architecture that made it pass the compliance review. Why air-gapped AI code review is different from everything else Most AI code review guides assume cloud connectivity. The model runs on the provider's infrastructure. The code is sent via API. The review comes back. The security question is about data handling policies, encryption in transit and vendor trust. In an air-gapped environment, none of that applies. The model runs on hardware you control. The code never leaves your network. There is no vendor to trust because there is no vendor in the loop. This sounds simpler from a security perspective. In some ways it is. In other ways it's significantly more complex because you're now responsible for the entire stack, model selection, quantisation, inference hardware, deployment, monitoring, updating and maintaining the system that in a cloud deployment is someone else's problem. Hardware requirements The hardware question is the fir

Learning software engineering in the past was a straight forward process, you learn the programming language, you build projects in your portfolio, apply to companies, get a job and life goes on. Doing that in the current times might be a little bit different, as when applying for jobs, you can see some new requirements other than your programming skills and portfolio project such as Prompt Engineering, Work with Agents, Claude Code, and others. You may ask yourself, what are those? And if I am new to Software Engineering, will that change my learning path? Lets discuss all this below. Software Engineering in the Past For a long time, the path into software engineering was clear. You picked a language, maybe Java, Python, or JavaScript. You spent a few months learning the syntax, then the fundamentals: data structures, algorithms, how a database works, how the web sends and receives data. After that, you built things such as A todo app, weather app, clone of a website you liked. These projects went into a portfolio, usually a GitHub profile and a simple personal site. Then you applied to companies, passed a technical interview, and started your first job, so the skills you needed were stable, if you learned React in 2018, React was still useful in 2021. Tools changed, frameworks came and went, but the core idea stayed the same: you write the code, you understand what you wrote, and you fix it when it breaks. When Did AI Start Becoming Something Required The shift did not happen in one day. It came in steps. The first step was autocomplete , around 2021, tools like GitHub Copilot started suggesting the next line of code while you typed. Most developers saw it as a nice helper, nothing more. It saved you from writing boilerplate, but you were still the one thinking. The second step was chat , when ChatGPT and Claude became popular, developers started using them to explain errors, review code, and write small functions. Still a helper, but a much stronger one. At this

Break reminders are your friend.

I regularly lose time to terminal muscle memory. I work across Windows and Unix-like shells, so I will remember the right command in the wrong environment, transpose a Git subcommand, or use a valid binary with an invalid subcommand. The fix is usually easy to find. The interruption is the costly part: stop, search, translate the answer back into the current shell, and try again. The idea is not new. There are projects that help with failed commands, as well as terminal-integrated LLMs such as aichat. However, I wanted something that does one thing well: fix commands in a seamless workflow powered by an LLM. I wanted a deliberately narrow tool: when a command fails, suggest the command I probably meant, let me inspect it, and run it only after confirmation. During a Fable promotion period, I used an LLM coding agent to see how far a well-scoped prompt could get me. With very little steering, it produced a usable Go prototype in roughly an hour. The project is now open source: nudge . The workflow I wanted The core interaction is intentionally small: Run a command as usual. If it fails, type fix (or bare nudge ) to get a suggested correction. Review it, then press Enter to run, e to edit it first, or n to cancel. The cheapest case is a plain typo, which never reaches a model at all: PS> git pshu git: 'pshu' is not a git command. See 'git --help'. PS> fix `git pshu` isn't a valid command. Did you mean: → git push (typo fix for `git pshu`) Run it? [Enter = yes / n = no / e = edit] The (typo fix for ...) label is the tool telling me it answered locally, in under 10 ms, without a network call. The cross-shell version of the same mistake is reaching for a binary that does not exist here. Because the binary is missing, the shell's command-not-found hook fires and I do not have to type anything at all: PS> printenv `printenv` isn't a valid command. Did you mean: → Get-ChildItem env: (list all environment variables) Run it? [Enter = yes / n = no / e = edit] The case that act

I would never go so far as to say there's no place for AI in music (I'm a fan of Holly Herndon, after all). But I generally find music made with generative AI to be offensively boring, especially the outputs of Suno. So I'm having a bit of a tough time processing the fact that […]

TL;DR: Passing a historical blockNumber to eth_call , eth_getBalance or eth_getLogs silently routes your request to the archive tier of hosted RPC providers. In our production metrics, archive calls cost on average 26.7x more compute units than the same call at latest . This post explains why, shows the exact client code pattern that triggers it, and gives you three Prometheus queries to measure your own archive exposure in under a minute. Full cross provider measurements are published in the OpenChainBench RPC benchmarks . Last week our RPC cost dashboard flagged an overage projection above four thousand dollars for a single billing cycle on a single provider. On paper, our services were doing normal eth_call operations. In practice, one small pattern buried in three separate indexers had multiplied our compute unit consumption by more than an order of magnitude, and nothing in the code review process had surfaced it. This post breaks down what an archive multiplier is, why it silently inflates blockchain RPC bills across every major hosted provider, and how to detect it in your own Prometheus stack before the next overage alert lands in Slack. What does "archive" mean at the Ethereum node level? Every request that reads the state of a smart contract, whether through eth_call , eth_getBalance , eth_getStorageAt , eth_getCode , or a batch of these, requires the RPC node to reconstruct the world state at a specific block height. Ethereum clients handle this in two modes. Full node mode. The state trie is kept in memory or on fast SSD for the tip of the chain plus a rolling window of recent blocks. On Geth default settings that window is 128 blocks deep. Any query targeting latest , pending , or a block within that window resolves in a few milliseconds against the current state. Archive node mode. The client preserves every intermediate state trie since genesis. Answering a query at a block from months or years ago requires reading historical trie data off disk and re

Over the past few months, I've been exploring how AI can move beyond chatbots and become an active part of business workflows. That journey led me to build SmartStock AI , an inventory management platform that combines modern web technologies with AI agents, Retrieval-Augmented Generation (RAG), demand forecasting, and automation. Instead of only tracking inventory, SmartStock AI helps businesses make proactive decisions. What SmartStock AI Can Do 🤖 Forecast future product demand 📦 Recommend purchasing decisions through AI agents 📚 Answer inventory-related questions using Hybrid RAG with source citations 📄 Process invoices using multimodal AI ⚡ Generate real-time inventory alerts 🔐 Secure the platform with JWT authentication and role-based access control Technology Stack Frontend React 19 Backend Django 5 Django REST Framework Database PostgreSQL pgvector AI LangChain Hybrid RAG AI Agents Prophet Forecasting Infrastructure Celery Redis Docker GitHub Actions What I Learned Building SmartStock AI taught me much more than integrating an LLM into an application. Some of the biggest lessons were: Designing AI features that solve real business problems. Building reliable agent workflows instead of simple chatbot interactions. Combining vector search with structured database queries. Managing asynchronous AI tasks using Celery and Redis. Creating production-ready APIs with Django REST Framework. Deploying and maintaining a modern full-stack application. Demo 🎥 YouTube Demo https://www.youtube.com/watch?v=DQJqs6bgE98 🌐 Live Demo https://smart-stock-dev.vercel.app/ Demo Account Email: viewer@smartstock.ai Password: Viewer123! 💻 GitHub Repository https://github.com/Eng-Ayman-Mohamed/SmartStock-AI Final Thoughts This project was developed as my graduation project during the Information Technology Institute (ITI) Full Stack Web & Generative AI Program. It was an incredible opportunity to explore AI engineering, backend architecture, and modern software development while buildin

Try asking this in your next team meeting: "Who on our team is the best at using AI?" There will probably be a moment of silence. Someone thinks of the person who uses the tools the most. Someone thinks of the person who's good with prompts. Someone thinks of whoever recently produced something impressive-looking with AI. But none of these answers feels convincing. There's a gut sense that something is off — and yet nobody knows what the right criterion would be. Nearly every company is standing in front of this wall right now. AI competence is rising to the top of hiring and performance criteria, and there's no yardstick to measure it with. This essay proposes one. The short answer is a set of five axes I call ABCD2. But before that, we need to fix the question itself. The very idea of "good at using AI" is wrong The title of this essay says "good at AI" — but that phrasing is itself the trap. The word "using" carries a hidden frame: AI is a tool, and the person is a user. It puts AI on the same shelf as Excel and Photoshop — something you get skilled at operating. On top of that frame, evaluation naturally becomes a measure of tool proficiency. How many features do you know? How fluently do you operate it? But AI is not a hammer. AI is an engine that judges and executes. It's less like a tool and more like a junior colleague you can delegate work to. And once you think of a junior colleague, the frame flips. Behind every junior who performs well, there is invariably a senior who makes them perform well — someone who explains the context of the work, lays out the criteria for judgment, makes it safe to ask about ambiguities, and verifies the output. That senior isn't "using" the junior. They are making the junior do the work well . AI is exactly the same. The person who truly gets the most out of AI is not the person who is good at using AI, but the person who helps AI do the work well. And what do they help with? Two things, broadly. One is knowledge building — pr