标签：#t

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

For a long time I ran product extraction off a database of custom selector configs. Hundreds of retailers, each with its own set of CSS selectors mapped field by field: price here, title there, image over there. It worked. It was also a treadmill. Every retailer that redesigned its frontend silently broke its config, and the maintenance tax grew with every retailer I added. Hundreds of configs is hundreds of things that can rot without telling you, usually right before someone downstream asks why a brand's prices went null. At some point I stopped feeding it. The scraping platform I ran at my last engagement fed a 20M+ product catalogue at millions of pages a day, and the thing that made that survivable wasn't better per-site config. It was leaning on almost no per-site config at all. The pattern Instead of one careful parser per site, run several cheap generic extractors on every page, in parallel: JSON-LD. A huge share of e-commerce pages ship a schema.org/Product block because Google rewards it. It has name, price, currency, availability, images. It's the closest thing to a public API hiding in the HTML. OpenGraph tags. og:title , og:image , product:price:amount . Lower quality than JSON-LD, but present on sites too lazy for structured data, because everyone wants pretty link previews. Microdata / RDFa. Older sites, still surprisingly common outside the US. Embedded state. __NEXT_DATA__ , window.__INITIAL_STATE__ and friends. A JSON blob the site's own frontend hydrates from. Heuristics. A price-shaped string near a currency symbol inside the main content region. The largest above-the-fold image. The h1 . Dumb, and dumb works more often than you'd think. None of these is reliable alone. OG price tags go stale, JSON-LD sometimes describes the wrong variant, heuristics grab the crossed-out "was" price. The trick is you don't pick an extractor. You pick per field, and you let agreement between sources carry the decision. Confidence merge Each extractor emits candida

A first-of-its-kind analysis found more than one in eight apps built for US service members carried foreign code—some from firms in nations the Pentagon designates as adversaries.

A cute little to-do list that celebrates with you Discussion | Link

The next time you apply for a job, AI may screen your résumé before any human sees it. But there’s good reason to question whether AI will judge you fairly. Researchers already know that LLMs pick up human biases from their training data. New research suggests that LLMs can also develop their own biases from…

Hey everyone. Just an update on the Salesforce Certification. I am planning on releasing articles...

Liquid syntax error: Variable '{{% raw %}' was not properly terminated with regexp: /\}\}/

Tu prueba pasó el lunes: misma entrada, mismo código y temperature=0 . El martes falló sin cambios en tu aplicación. La aserción esperaba una cadena exacta, pero el modelo devolvió la misma respuesta con una redacción ligeramente distinta. El agente funciona; tu suite de pruebas no. Prueba Apidog hoy Este es el coste de probar sistemas que llaman a modelos de lenguaje. Incluso con temperatura cero, no obtendrás salidas idénticas byte a byte entre ejecuciones. En lugar de probar la redacción exacta, prueba el contrato: estructura, campos, rangos y efectos esperados. Este artículo profundiza en el tercer modo de fallo de nuestra guía sobre por qué los agentes de IA fallan en producción . Por qué temperature=0 no significa determinismo La temperatura controla cómo el modelo selecciona el siguiente token. Con temperature=0 , el modelo elige el token más probable, lo que parece reproducible. Sin embargo, esa configuración no garantiza resultados idénticos. La causa está debajo del modelo: Las operaciones de punto flotante en GPU no son asociativas: el orden de las sumas puede modificar mínimamente el resultado. Esa diferencia puede cambiar cuál token queda en primer lugar. El orden de ejecución puede depender de cómo el proveedor agrupa solicitudes, del hardware, de la región o de la versión del kernel. Los proveedores pueden cambiar GPUs, bibliotecas de inferencia o cuantización de pesos. Una discusión extensa en vLLM explica por qué una semilla fija y temperature=0 no bastan para lograr reproducibilidad bit a bit. La conclusión práctica es simple: el determinismo es una propiedad de toda la pila de servicio, no una opción de tu solicitud. Tu prueba debe aceptar variaciones de redacción cuando el significado y el contrato siguen siendo correctos. Por qué las aserciones de cadena exacta vuelven inestable tu suite Esta prueba es frágil: assert response == " Your order total is $42.00. " Puede pasar hoy y fallar mañana si el modelo responde: Your total comes to $42.00. La

Your test passed on Monday. Same input, same code, temperature=0 . On Tuesday it failed, and you changed nothing. The assertion expected an exact string, but the model returned the same answer with slightly different wording. The agent is fine; the test is not. Try Apidog today This is the cost of testing language-model output. Even at temperature=0 , you cannot rely on byte-identical responses across runs. Instead of testing exact wording, test the API contract: structure, required fields, valid ranges, tool-call payloads, and safety constraints. This is a deeper look at failure mode three in our guide to why AI agents break in production . Why temperature=0 does not mean deterministic Temperature controls token sampling. At zero, the model selects the most probable next token, which sounds reproducible. In practice, the serving stack introduces variation. GPU floating-point math is not associative: adding the same values in a different order can produce small numerical differences. Those differences can change which token ranks highest. Once one token differs, every token after it can diverge. The order of operations can vary based on: Request batching GPU hardware Inference kernels Provider routing and regions Model-serving library versions Weight quantization changes Providers can also update infrastructure without changing your API request. As this vLLM discussion explains, a fixed seed and temperature=0 are not enough for bitwise reproducibility. Determinism is a property of the entire serving stack, not a request parameter. Design tests accordingly. Why exact-string assertions make tests flaky This assertion is fragile: assert . equal ( response , " Your order total is $42.00. " ); It fails if the model returns a correct variation: Your total comes to $42.00. The failure does not indicate a product regression. It indicates that the test is coupled to wording that is expected to vary. Flaky tests create a predictable failure pattern: Developers rerun the suite

Running a production database on a minimal two-core PostgreSQL instance presents unique engineering challenges. In resource-constrained environments, default configurations quickly lead to CPU exhaustion, memory thrashing, and high latency. To maintain stable performance, you must aggressively manage resource allocation and optimize query execution plans. The first critical area is connection management. PostgreSQL allocates a separate operating system process for each connection. On a two-core machine, allowing hundreds of concurrent connections will cause severe context switching overhead, degrading performance significantly. You should restrict maximum connections to a low double-digit number and implement a connection pooler like PgBouncer. A pooler ensures that incoming traffic queuing happens outside the database engine, allowing the CPU to focus on executing active queries rather than managing process states. Memory allocation parameters require precise tuning on limited hardware. The shared buffers parameter, which dictates how much memory PostgreSQL uses for caching data, should typically be set to twenty-five percent of total system RAM. However, work memory is where many developers run into trouble. The work memory setting determines the amount of memory used by internal sort operations and hash tables before writing to temporary disk files. Since this memory is allocated per query operation, a complex query with multiple joins can consume many times the configured value. On a two-core instance with limited RAM, setting this value too high can trigger the out-of-memory killer, while setting it too low forces slow disk-based sorting. You must analyze your heaviest queries and find a balanced value that prevents disk thrashing without exhausting system memory. Query optimization becomes a daily necessity when compute resources are scarce. You must regularly inspect query execution plans using the explain analyze command to identify sequential scans on large

We’ve all been there: you are busy coding, and suddenly you get a scary error saying your code conflicts with a teammate's work. Git won't let you pull their changes because it's afraid of overwriting your unsaved files. Instead of panicking, I used git stash. This command acts like a temporary clipboard—it safely hides your current work away so your workspace becomes completely clean. With a clean screen, I was able to safely update the project, fix the conflicting lines of code by hand, and make sure everything compiled perfectly. Once the team's updates were safely pushed, I ran git stash pop to bring my hidden work right back. Git wasn't trying to break my project; it was just protecting my code. Next time you get stuck, remember the golden rule: Stash your work, fix the conflict, and pop it back!

Most people working in offices never think about why the temperature stays comfortable throughout the day. The cooling adjusts automatically. Fresh air increases when occupancy rises. Fans start and stop without anyone touching a switch. Behind all of this is a device that most building occupants have never heard of: the DDC Controller. The Hidden Computer Inside Every Modern Building Walk into a mechanical room and you'll find equipment everywhere: Air Handling Units (AHUs) Chillers Pumps Cooling Towers VAV Boxes All these systems need coordination. If the supply air temperature rises above its target, something has to react. If occupancy increases, fresh air must increase. If a fan trips, alarms must be generated. This is where a DDC controller comes in. Think of it as a small industrial computer dedicated to one job: keeping a building running efficiently. A Typical Day in the Life of a DDC Controller Imagine an AHU supplying air to an office floor. At 9:00 AM employees begin arriving. The return air temperature starts increasing. The DDC controller notices this through a temperature sensor. Within seconds it: Reads the sensor value Compares it against the setpoint Calculates the cooling demand Adjusts the chilled water valve Verifies fan operation Repeats the process No operator is required. No manual intervention is needed. The controller quietly performs these calculations all day. Why Not Just Use a PLC? This is one of the most common questions from engineers entering building automation. PLCs and DDC controllers are both programmable devices. However, they were designed for different worlds. A PLC excels at: Manufacturing lines Packaging machines Process control High-speed sequencing A DDC controller excels at: HVAC control Energy optimization Occupancy schedules Comfort management BACnet communication Both can control equipment. The difference is what they were originally built for. The Four Signals Every BMS Engineer Learns First If you're new to building

Streaming LLM responses in TypeScript: SSE, ReadableStream, and the React 19 useChat hook. The first time I wired an LLM response that streamed token by token instead of arriving as one lump after 4 seconds, I shipped it to production the same afternoon. The difference in perceived speed is that obvious to anyone who has used ChatGPT and then tried a non-streaming competitor. Users will wait 8 seconds if they can see the cursor moving. They will not wait 4 seconds for a blank screen. This tutorial walks the full stack from scratch. By the end you have a working Next.js API route that streams from an LLM over Server-Sent Events, a frontend that parses the stream manually with ReadableStream , and then the same UI rebuilt with the Vercel AI SDK's useChat hook so you can see what the abstraction actually buys you. TL;DR Layer What you build Key API Next.js route Streams LLM output as SSE streamText + toUIMessageStreamResponse Vanilla client Parses the stream by hand ReadableStream , TextDecoderStream React 19 client Managed state + cancellation useChat from ai/react Edge cases Backpressure, cancel, tool chunks AbortController , partial JSON guard 1. Why streaming matters A standard fetch returns after the entire response body is ready. For short completions, that is fine. For anything over 100 tokens, users see a spinner, then a wall of text, then confusion about whether the app is fast or slow. Streaming changes the shape of that experience. The first token lands in under 300ms for most hosted models. The user starts reading while the model is still writing. Perceived latency drops by 60 to 70 percent even if the total time to complete the response does not change. Cost transparency is the second reason to care. When you stream, you count tokens as they arrive. If your route has a budget ceiling and the response is going to blow past it, you can cut the stream at 800 tokens without ever waiting for the full completion. That cut is not possible with a blocking call. 2.

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Originally published on the MyTreda engineering blog: read here A support message described what looked like one confusing mobile bug. It turned out to be two independent ones — a cross-site cookie getting blocked by Safari's ITP, and a React Hook Form autofill desync — that just happened to both only show up on mobile. Full breakdown, code, and fixes below.

Artificial intelligence is changing the way websites are planned, built, managed, and improved. WordPress developers now have access to tools that can help with coding, content creation, customer support, automation, analytics, and search engine optimization. However, using an AI plugin does not automatically make someone an AI-ready developer. A genuinely AI-ready WordPress developer understands how to combine technical experience, business thinking, automation, and human judgment. The goal is not to add AI everywhere. The goal is to use it where it solves a real problem. AI Should Solve a Clear Business Problem Many businesses make the mistake of choosing an AI tool before deciding what they actually need. A better approach starts with a practical challenge. For example, a company may want to: Respond to customer questions more quickly Organize website enquiries Improve WooCommerce product recommendations Automate repetitive administrative work Connect website forms with a CRM Generate content drafts Analyze customer behaviour Improve internal support processes An experienced developer will first study the workflow, the expected result, the available data, and the possible risks. Only after that should a suitable plugin, API, automation platform, or custom solution be selected. This approach prevents businesses from spending money on features that look impressive but provide little value. AI-Generated Code Still Needs Human Review AI coding tools can produce functions, snippets, plugin ideas, and debugging suggestions within seconds. That speed is useful, especially when a developer is working with repetitive tasks or unfamiliar code. The danger is that AI-generated code may look correct while containing hidden problems. It can include: Outdated WordPress functions Weak security practices Plugin compatibility issues Poor database queries Unnecessary scripts Incorrect assumptions Performance problems That is why generated code should never be added directly to a li

Here's a problem every agent builder eventually runs into: your assistant needs to be good at a lot of things. Writing SQL. Composing emails. Debugging code. Maybe more later. So where do all those instructions go? The obvious answer is: cram them all into the system prompt. But that "obvious" answer causes real problems as your agent grows. In this post, I'll walk through those problems first, then show how the skills pattern in LangChain fixes them, using a small working example. The problem: one giant prompt Let's say you want a single assistant that can write SQL, draft emails, and help debug code. The naive approach is one big system prompt: You are a helpful assistant. When writing SQL: - Write queries using only SELECT, INSERT, UPDATE, DELETE - Always use parameterized queries to prevent SQL injection - Format SQL keywords in UPPERCASE - Include brief comments for complex queries When writing emails: - Keep emails concise and professional - Use clear subject lines - Structure: greeting -> purpose -> details -> call to action -> sign-off - Adjust tone based on context When debugging: - First reproduce the error - Check logs and error messages - Isolate the root cause - Suggest a fix with explanation ... This works fine for three domains. But keep adding more — legal reviewing, data analysis, customer support scripts, code review standards — and you run into a few concrete issues: 1. Prompt bloat. Every single request pays the token cost of every domain's instructions, even if the user only asked about SQL. That's slower and more expensive for no benefit. 2. Instructions start to blur together. When the SQL rules, the email rules, and the debugging rules all sit in the same block of text, the model has to sift through everything at once. It's easy for instructions from one domain to bleed into another, or for the model to lose track of which rule applies where. 3. It doesn't scale. Every time you want to add a new specialty, you're editing one long, increasingl

If you’ve spent any time recently writing detailed product requirement documents or meticulously...

Meme Monday! Today's cover image comes from the last thread. DEV is an inclusive space! Humor in...

Open weights 975B multimodal model built for fine-tuning Discussion | Link