标签：#LLM

Beyond Size: The Three Pillars of Test-Time Scaling in Large Language Models The narrative of artificial intelligence for the last decade has been dominated by a single, powerful trend: scaling. From the early days of AlexNet to the massive clusters powering GPT-4, the formula seemed simple—more data and more parameters lead to better performance. This paradigm, famously codified as the "Scaling Laws," suggested that we could predict model improvements simply by looking at the amount of compute poured into the pre-training phase. However, as the industry pushes against the boundaries of available high-quality data and the physical limits of hardware, a new dimension of scaling is emerging. It isn't about how large the model is, but how long it "thinks" before it speaks. This shift toward "test-time scaling" marks a transition from static intelligence to dynamic reasoning. Instead of relying solely on the patterns learned during training, models are now being equipped with the computational budget to explore, verify, and refine their answers at the point of inference. While the concept was popularized by the release of models like OpenAI’s o1 series , the underlying mechanics remained somewhat opaque. A recent comprehensive study by Hariri et al. (2026), titled " Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility ", provides a much-needed formal framework for understanding this new frontier. The Three Regimes of Inference Compute The core contribution of the Hariri et al. paper is the formalization of test-time scaling into three distinct structural regimes. Rather than treating all "extra compute" as a single scalar budget, the authors map how compute is allocated across the implicit prefix tree of an autoregressive model. 1. Single-Trajectory Sequential Scaling This is the most familiar regime, often associated with Chain-of-Thought (CoT) prompting. In this mode, the model generates a single sequence of tokens. Compute is scaled

Four days after I published a piece arguing LLMs can't make the jump, OpenAI announced that an...

A slow AI feature does not feel smart. It feels broken. That is the uncomfortable truth many AI SaaS builders hit after the demo works. The prototype answers well, the agent can call tools, and the RAG pipeline looks impressive. Then real users arrive. Prompts get longer. Queues form. Streaming starts late. One tenant uploads huge documents. Another runs bulk jobs at noon. Suddenly the same workflow that felt magical in testing feels like a spinner with an invoice attached. The fix is not simply “use a faster model.” You need an LLM latency budget : a small set of rules that says how fast each AI workflow must feel, how many tokens it can spend, when to stream, when to cache, when to route to another model, and when to stop before cost and latency drift together. This guide is for solo SaaS developers, micro SaaS builders, and AI SaaS teams shipping production features with LLM APIs, RAG, agents, or self-hosted models. Why latency budgets matter now AI platform news points in the same direction: builders are moving from chat demos to production workflows. Agent tools, web context APIs, voice agents, coding assistants, and RAG platforms are all getting more capable. At the same time, inference cost and reliability are under pressure. Latency is now a product metric. Inference efficiency is becoming a business metric. Yet many articles stop at TTFT, TPOT, quantization, batching, or model serving. Fewer show how a SaaS builder turns those ideas into a product-level budget with code, dashboards, fallbacks, and customer-safe limits. The simple model: TTFT, TPOT, and total time You do not need a PhD in serving systems to start. Track three numbers. Time to First Token Time to First Token (TTFT) is the delay between the user action and the first streamed token. It includes network time, queue time, provider overhead, tool setup, retrieval, and the model’s prefill phase. High TTFT is why a chat box feels dead. Time Per Output Token Time Per Output Token (TPOT) is the averag

Cada vez más gente arranca la búsqueda de un proveedor preguntándole a un modelo en vez de a un buscador. Y no pide diez opciones para comparar: pide una recomendación y recibe dos o tres nombres. Si tu empresa no está ahí, no quedaste octava. No estás en la respuesta. La pregunta que sigue es obvia: cuánto tarda en cambiar eso. Pero antes hay un problema más aburrido y más importante, que es cómo se mide. Lo escribo porque es la parte que casi nunca se cuenta y es donde se rompen los informes. Una captura de pantalla no es una medición Es el error más común y el más difícil de discutir, porque la captura parece prueba. La respuesta de una app conversacional depende del historial de la cuenta, de la sesión, del ruteo interno del proveedor, de si esa consulta activó búsqueda web o no, y de la región desde donde se pregunta. Dos personas preguntando lo mismo el mismo día reciben respuestas distintas. La misma persona preguntando dos veces también. O sea: la salida no es determinista y el instrumento no es estable. Una captura te dice qué pasó una vez, en un contexto que no podés reconstruir. Como métrica de seguimiento no sirve para nada. Lo que sí sirve es una serie: la misma consulta, literal, contra el mismo motor, con el mismo criterio de clasificación, repetida en el tiempo. El valor absoluto de un punto importa poco. Lo que importa es la diferencia entre puntos. Fijar el texto de la consulta, no la etiqueta Este es un bug de proceso que da resultados verosímiles y falsos. Si guardás en la planilla una etiqueta como "consulta de chatbot" en vez del texto exacto que preguntaste, dentro de dos meses nadie se acuerda del wording. Y el wording cambia el resultado: preguntar "quién hace X en Argentina" y "mejores empresas de X en Argentina" devuelven listas distintas. Cuando el texto se corre entre rondas, la serie deja de ser comparable, pero el gráfico sigue dibujándose igual de lindo. Guardá el string literal, versionado. Si tenés que cambiar una consulta, empezá u

How EvalPort's Grader System Works When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own way of scoring LLM outputs — DeepEval uses metric classes, Promptfoo uses assertion objects, Inspect AI uses solver functions. We needed a system expressive enough to cover 90%+ of real-world eval needs, but simple enough that any framework could implement it. The result: 11 grader types that carry their own semantics. A grader isn't just a name — it specifies its parameters, its model, its threshold. An eval suite is self-describing. The 11 Grader Types exact_match — Compare output to expected output, optionally ignoring case. contains — Check if the output contains a substring. regex — Match against a regular expression. semantic_similarity — Embed output and expected output, compare cosine similarity against a threshold. llm_judge — Use an LLM to evaluate the output against a prompt template. The most powerful grader. json_schema — Validate that the output is valid JSON matching a JSON Schema. json_path — Extract a value from JSON output using a JSONPath expression, then compare it. code — Run a function to evaluate the output. human — Defer to human review. model_graded — Compare the output to a reference answer using a model. custom — Escape hatch for graders not covered by built-in types. How Graders Connect to Test Cases A test case references graders by ID. Multiple graders can evaluate the same test case. The ResultSet records each grader's score separately. Why This Design Works Self-describing: An eval suite carries everything a framework needs to execute it. Framework-agnostic: Any framework can implement any subset of grader types. Extensible: The custom type lets frameworks bring their own graders. Comparable: Results from different frameworks use the same grader IDs. Try It pip install evalport-sdk npm install evalport-sdk Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md Repo: https://gith

Most business prediction problems do not arrive as prose. They arrive as rows: account attributes, transactions, sensor readings, test results, and a target column. For this kind of data, gradient-boosted trees and other conventional methods remain hard to displace. A new paper, Why Large Language Models Fail at Tabular Prediction , asks a much more useful question than “can an LLM classify a table?”: what, specifically, breaks as the task becomes more like ordinary tabular machine learning? The answer from the authors’ controlled experiments is input dimensionality. Their result matters because it separates a real limitation from several explanations that sound plausible but did not hold up in their tests. The experiment was about prediction, not table chat The paper evaluates frontier LLMs in a pure inference setup: a model receives labeled examples and must predict labels for new rows in a single generation pass. There is no fine-tuning, retrieval pipeline, tool calling, or agent loop to compensate for the base model. This is deliberately narrow. It asks whether a general-purpose language model can act as a direct tabular learner. Across 31 benchmark datasets, the authors compare nine methods and 252 configured classical models. That scope is important: a weak result on one CSV is easy to explain away as prompt design or a quirky dataset. A consistent trend across many tasks is harder to dismiss. The headline is not simply that LLMs lose to established tabular baselines. It is that their accuracy declines as the number of input dimensions grows, while the classical baselines in the study stay stable or improve. The paper therefore treats dimensionality as the central failure mode rather than an incidental property of difficult datasets. Four popular explanations did not survive testing There are several standard reasons developers give for poor LLM performance on tables. The researchers turn these into falsifiable hypotheses. “The classes overlap too much.” If th

I built an orchestration skill for Claude Code that delegated everything to subagents. It worked. It also cost somewhere on the order of 1-2 million Opus tokens per task - including tasks whose final diff was a handful of lines. Nothing was broken. Every individual decision was defensible. Three modest multipliers stacked, and then the whole stack ran on every single request. This is the postmortem, the redesign, and the enforcement layer I should have written first. v1: pure delegation The design goal was context hygiene. The main session gets polluted fast - it accumulates file contents, tool output, and dead ends, and its judgment degrades as the window fills. So: don't let it do any work. Make it a coordinator, and give every unit of real work a fresh context. That produced four rules: A hard gate. The main session was forbidden from reading, editing, or running anything itself. Every action went through a subagent. A fixed 5-phase pipeline on every task: Plan → Approve → Execute → Review → Report. Fresh subagents per phase. No reuse. Each phase got clean context by construction. Mandated reviewers with "loop until clean." A review phase that re-ran until it found nothing. And the trigger was broad - essentially any actionable request. "do this," "implement," "fix," "build," "change." Read those four rules again with a cost lens instead of a correctness lens. That is the whole postmortem. The three multipliers 1. The dispatch schema made model optional The subagent dispatch tool takes a model parameter. My skill never set it. Omitted, it inherits from the parent session - which was Opus 4.8. So every subagent, including the ones whose entire job was "read this file and summarize it," ran on the most expensive tier available. Here's what that actually costs at list prices: Model Input $/MTok Output $/MTok vs. Opus Claude Opus 4.8 ( claude-opus-4-8 ) $5.00 $25.00 1× Claude Sonnet 4.6 ( claude-sonnet-4-6 ) $3.00 $15.00 0.6× Claude Haiku 4.5 ( claude-haiku-4-5 ) $1.

A product can look healthy while its AI feature quietly loses money on every successful user action. The demo feels fast, the answers look useful, and usage is growing. Then the bill lands, and nobody can explain which workflow, tenant, prompt, model route, or retry loop consumed the margin. That is the practical value of inference efficiency ratio . It gives builders a simple question to answer before scaling an AI workflow: for every dollar spent on production inference, how much product value did the system create? This article shows how to instrument that answer without turning your codebase into a finance spreadsheet. Working definition: Inference Efficiency Ratio = AI-attributed product revenue / production inference cost You do not need a huge finance team to use it. You need clean events, honest cost attribution, and a dashboard that makes bad unit economics visible early. Why builders are talking about inference efficiency now Recent AI news has a clear pattern: agents are doing more real work, open-weight models are pushing prices down, and teams are moving from demos into production operations. At the same time, builders are asking harder questions about cost, security, reliability, and whether AI workflows can survive real customer usage. The current signals are hard to miss: Hacker News discussions are focused on open-source AI infrastructure, cloud coding agents, production access, and model price-performance. Developer content is moving from "try this model" toward "operate this workflow safely and cheaply." AI cost writing is shifting from token price alone to product-level unit economics. Multi-agent systems, web context pipelines, and voice agents are increasing the number of hidden model calls per user action. The gap: many articles explain token counting, caching, or model routing. Fewer show how to connect those details to product margin in a way a solo builder can implement. That is the angle here. What inference efficiency ratio actually measu

Give an LLM a fish, feed it for a millisecond. Teach an LLM to fish, and become Master of the Lake! I've just published "Master of the Lake" - available for listening EXCLUSIVELY at tvox.online/books/1 . (Nudge, nudge, NoStarch...) This work represents the culmination of years developing Contract Style Comments (CSC) as a practical interface for governing AI agents in what I'm calling the "Agentic Epoch" - an era where AI agents are no longer passive tools but active participants in our systems. The Interface Problem We've spent decades refining how humans interact with computers: command lines, GUIs, touch interfaces, voice assistants. But we've largely ignored the critical interface problem of our time: how do humans govern AI agents? Most AI tooling focuses on making agents more capable - better at generating code, more creative in design, more persuasive in writing. But capability without governance creates dangerous systems that appear functional while silently drifting from intent. CSC provides the missing interface layer: a structured way for humans to specify, verify, and maintain governance over agent behavior. Beyond Prompts: The CSC Interface Prompt engineering treats agents like fickle genies - rub the lamp the right way and maybe you'll get what you want. But this approach fundamentally misunderstands the agentic relationship. CSC shifts us from: Prompting (hoping the agent understands) To: Contracting (explicitly defining what the agent must uphold) The interface consists of three interconnected files: contract.md - The operational interface: what the agent must do why.md - The explanatory interface: why those requirements exist invariant.md - The boundary interface: what can never change This isn't just documentation - it's a verifiable interface that agents can check against continuously, making the alignment gap visible before it causes harm. System Silent-Death: Why Interfaces Matter We used to fear system crashes - honest failures that clearly ind

Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model. The two phases of inference Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large. The machines, and how they were measured Machine CPU / RAM GPU (VRAM) Storage (read) Prefill (tok/s) Gen (tok/s) Load (18 GB) Primary desktop 5950X / ~80 GB DDR4 RX 6900XT (16 GB) NVMe (~2.1 GB/s) 360 18.3 8.4s Secondary box 5600G / 32 GB DDR4 GTX 1060 (6 GB) SATA SSD (~0.35 GB/s) 253 17.1 50.6s Laptop 8840U / 32 GB DDR5 Radeon 780M (none) NVMe (~2.4 GB/s) 20 10.0 7.5s All inference figures come from a controlled run: the same model (Gemma 4 26B, 18 GB) on each machine, a unique random prefix per prompt to defeat caching, a fixed 8,192-token context, warm, on an identical ~6,855-token prompt (generation timed over a 200-token output). Two things stand out. Prefill varies about eighteen-fold across the machines (360 to 20 tok/s) while generation varies less than twofold (18.3 to 10.0), and prefill is what dominates large-prompt workloads — so a machine can look fine on generation yet be useless in practice. Model-load time, separately, is set by storage rather than compute: the secondary box's budget SATA SSDs load the 18 GB model in 50 seconds against eight on NVMe, which turns a cold request into a minute-long stall. Secondary box Request time Warm (model resident) ~4s Cold (model reload) ~54s If the model is allowed to unload between calls, every call silently pays that reload — a real source of intermittent timeouts. The fix is

Every once in a while an AI announcement lands that isn't about a chat UI or a new benchmark, but about the actual substance of what these systems can now do. OpenAI's announcement of ten new results in mathematics and theoretical computer science — produced by an internal version of Astra, their next major model — is one of those moments. Here's what happened, why it matters beyond the math community, and where the honest caveats are. The results The ten problems span high-dimensional geometry, coding theory, arithmetic circuit complexity, group theory, operator algebras, quantum complexity, lattice cryptography, and extremal combinatorics. Highlights include: Non-sofic groups — a construction establishing their existence, addressing a central open question in group theory. Connes's rigidity conjecture — a disproof of a longstanding conjecture about von Neumann algebras. Quantum parallel repetition — an exponential parallel repetition theorem for general two-player quantum games. Multicolor Ramsey numbers — a superexponential lower bound, resolving Erdős problem 183. Closest vector problem — polynomial-factor hardness of approximation, a foundational lattice question tied to post-quantum cryptography. Each argument was prepared into a manuscript by humans working with the model, then formalized by the model into a Lean certificate (the proofs are public on GitHub). OpenAI also released the model's narration of its own thinking process for each solution. The price tag that reframes everything The most striking number in the announcement isn't the math — it's the cost. The total tokens needed to find these solutions would cost roughly $2,000 at Sol API rates . Think about that for a second. Two thousand dollars of compute to resolve open problems that mathematicians have worked on for decades. Some of these (like non-sofic groups) have been open for over a decade of intense effort. We're not talking about a moonshot lab budget — we're talking about the price of a mid

XML Tagging in Prompts: The Secret to Getting Better Output from Claude and GPT A simple structuring trick that turns messy, unpredictable LLM outputs into clean, reliable ones. If you've spent any time writing prompts for Claude, GPT, or any other large language model, you've probably hit this wall: your prompt works fine for a simple ask, but the moment you pack in multiple instructions — some context, a few examples, formatting rules, and the actual task — the model starts mixing things up. It answers the wrong part of the question. It ignores your formatting instructions. It treats your example output as part of the actual task. The fix is almost embarrassingly simple: wrap your prompt sections in XML tags. Why XML Tags Work So Well LLMs are trained on enormous amounts of code, documentation, and markup. XML (and HTML) syntax is deeply embedded in that training data, which means models are very good at recognizing where one tagged section ends and another begins. Unlike plain paragraphs — where the boundary between "here's my context" and "here's my instruction" is fuzzy — a tag creates an unambiguous boundary. Anthropic actually recommends this explicitly for Claude: wrapping distinct parts of a prompt (instructions, context, examples, output format) in tags like <instructions> , <context> , <example> , and <output_format> measurably improves consistency, especially in longer or more complex prompts. Think of it like the difference between handing someone a wall of text versus handing them a form with labeled fields. Both contain the same information, but one is far easier to parse correctly — for a human, and for a model. A Before-and-After Example Without tags: Summarize the article below in 3 bullet points. Keep it under 50 words. Use a neutral tone. Here's an example of the style I want: "- Company X raised $10M in Series A funding." Now here's the article: [long article text] The model has to guess where the instructions end and the article begins — and wi

I’ve been building AI apps with coding agents for a while. Lately, I’ve been experimenting with evals too. The app in this example mostly worked. That was the problem. The bug I built a small support-triage app for a fictional shipment-tracking company. A customer sends a support ticket, and the app decides what it is about, how urgent it is, and whether a human needs to respond. A real outage should be escalated. But this ticket was different: “URGENT need key rotation now” The customer was asking how to rotate their own API key before a security review. The app classified it as a security incident and escalated it to a human. That was wrong. The policy said normal key rotation was a self-service how-to request. Nothing crashed. The app returned valid JSON. The fields all contained allowed values. The behavior was still wrong. Why clicking around wasn’t enough I could test a few tickets manually and convince myself the app worked. But after changing the prompt, what would I actually know? Would the outage case still escalate? Would normal how-to questions stay in the normal queue? Would another API-key question behave differently? I didn’t want to change the prompt and simply hope for the best. I wanted a set of cases I could run again. Adding DeepEval with Cursor I installed the DeepEval agent skill: npx skills add confident-ai/deepeval --skill "deepeval" Then I asked Cursor to add evals to the app: This app sometimes treats normal support questions like emergencies and sends them to a human. Add DeepEval so I can test this using the tickets and policy already in the repo. I am new to evals, so use the simplest setup DeepEval already provides, explain what you create, and ask me anything you need. Run the app as it is first and show me what fails. Do not fix it yet. Cursor already had the app, tickets, and policy, so it went straight to creating the baseline. Goldens are the checklist The first useful artifact was a JSON dataset. Each golden contained: the custome

AI agents are everywhere in 2026. Most of them can answer questions, generate code, or automate simple workflows. But once you ask them to interact with a real computer—browsers, desktop applications, terminals, files, and external services—things quickly become unreliable. That was the motivation behind HeyAgent . The Problem Most autonomous agents fail for one of three reasons: They declare success before the task is actually finished. They lose context during long, multi-step workflows. They aren't designed to work with a real desktop environment. I wanted to build an agent that behaves more like a real assistant instead of just another LLM wrapper. What HeyAgent Does HeyAgent is an open-source autonomous AI agent for computer control and workflow automation. It can: 🖥️ Control desktop applications 🌐 Work inside browsers 📂 Read and manage files 💻 Execute terminal commands 🔗 Connect with external services 📱 Be controlled through CLI, Desktop UI, or Telegram 🧠 Plan and execute multi-step workflows ✅ Verify results before marking tasks as completed Instead of blindly executing prompts, the agent plans, executes, validates the outcome, and only then reports success. Reducing False Task Completion One of the biggest problems I noticed in existing AI agents is false task completion. Many agents click a button, assume everything worked, and immediately report success. In reality, something may have failed several steps earlier. HeyAgent performs additional verification after critical actions to reduce false positives and improve reliability during long-running workflows. Built with AWS Support HeyAgent has been significantly accelerated thanks to the support of AWS. AWS has provided the project with cloud infrastructure, GPU computing resources, and access to modern AI services that made rapid experimentation possible throughout development. From running GPU workloads to evaluating different LLMs and AI models, AWS has been an important part of the engineering process.

This essay originally appeared in Foreign Policy . Earlier this month, two of OpenAI’s models broke out of their containment sandbox and attacked another AI company. The story is kind of wild . OpenAI was running security tests on two of its models: GPT-5.6 Sol and an unreleased model that is almost certainly GPT-6. In particular, it was running the ExploitGym benchmark, which measures how good a model is at turning security vulnerabilities into working exploits: basically, offensive cyberattacks. Since these were internal tests, OpenAI locked those models in a secure sandbox that denied them access to the internet. But it was running the models without any safety filters that would prevent them from offensive cyber-actions. That meant that there was nothing to prevent the models from trying to ...

Prompts rot silently. I swap a model, tweak a system prompt, add a tool, and everything still runs. No exception is thrown, no test goes red, the JSON still parses. The output is just quietly worse, and I usually find out from a user rather than from CI. Unit tests are the wrong instrument here because there is nothing to catch: the failure mode is not a crash, it is a drop in quality. So I built evalgate , a small TypeScript tool that treats prompt and agent quality like a build artifact. You write a declarative eval suite, evalgate runs it, scores it, stores a baseline, and on every pull request it re-runs the suite, computes the quality delta against the base branch, and fails the build when the score regresses. Then it posts the delta table as a PR comment. The core idea The important design decision is what question CI is allowed to ask. "Is this prompt good?" is subjective and unwinnable in an automated gate. "Is this worse than it was on main?" is objective and answerable. evalgate is built around that second question. You capture a baseline once, and from then on every change is judged as a delta against it, not against some absolute notion of goodness. The second decision was that the whole thing has to run with zero API keys. evalgate ships a deterministic mock provider, so you can run a suite, save a baseline, compare runs, and execute the full test suite completely offline. The project itself has 67 tests and none of them touch the network. Every feature has to work in mock mode before it counts as done. How it works A suite is a YAML (or JSON) file that lives in version control next to the code it checks. Each case has an input, an expected reference value, and one or more scorers. Here is a minimal one: name : my-agent provider : mock # works with no API key threshold : 0.9 # mean score required to pass cases : - id : greeting input : prompt : | Reply with the standard greeting. exactly: Hi there! How can I help you today? expected : " Hi there! How ca

If you're running local models through Ollama and tired of Python's overhead, check out langchain-rust . It's a full LLM framework in pure Rust that works great with local models: Ollama support — first-class integration with tool calling, vision, and streaming 9 vector store backends — InMemory, SQLite, Qdrant, ChromaDB, Redis, PGVector, MongoDB, Pinecone, FileVectorStore BM25 keyword search — with Chinese/English tokenization, no external dependency Hybrid retrieval — BM25 + Vector with RRF fusion for better recall GraphRAG — Knowledge graph construction + community detection, all local CorrectiveRAG — Self-correcting retrieval with hallucination detection Code Interpreter — LocalSandbox (subprocess), E2B cloud, or WASM sandbox LocalEmbeddings — Run embeddings without calling an API Plus: LangGraph workflows, MCP client/server, 7 memory types, guardrails, and 12+ built-in tools. Single binary, no virtualenv, no pip conflicts. Just cargo add langchainrust and go. GitHub: https://github.com/atliliw/langchainrust Docs: https://docs.rs/langchainrust

Most AI applications wait for the model to generate the complete answer before showing anything to the user. For short answers, that may be acceptable. For longer responses, it can make the application feel slow—even when the model is already generating tokens. Streaming solves this by displaying each part of the response as soon as it arrives. The non-streaming version A standard OpenAI-compatible request may look like this: import os from openai import OpenAI client = OpenAI ( api_key = os . environ [ " AI_API_KEY " ], base_url = os . environ [ " AI_BASE_URL " ], ) response = client . chat . completions . create ( model = os . environ [ " AI_MODEL " ], messages = [ { " role " : " user " , " content " : " Explain API gateways in three sentences. " , } ], ) print ( response . choices [ 0 ]. message . content ) This works, but nothing is printed until the complete response has arrived. Stream the response Enable streaming by adding stream=True : stream = client . chat . completions . create ( model = os . environ [ " AI_MODEL " ], messages = [ { " role " : " user " , " content " : " Explain API gateways in three sentences. " , } ], stream = True , ) The request now returns a sequence of chunks instead of one completed response. Loop through those chunks and print the available content: for chunk in stream : content = chunk . choices [ 0 ]. delta . content if content : print ( content , end = "" , flush = True ) print () The user can now see the answer while it is being generated. Complete example import os from openai import OpenAI client = OpenAI ( api_key = os . environ [ " AI_API_KEY " ], base_url = os . environ [ " AI_BASE_URL " ], ) stream = client . chat . completions . create ( model = os . environ [ " AI_MODEL " ], messages = [ { " role " : " user " , " content " : " Explain API gateways in three sentences. " , } ], stream = True , ) for chunk in stream : content = chunk . choices [ 0 ]. delta . content if content : print ( content , end = "" , flush = True )

Introduction Artificial Intelligence has moved from being an experimental technology to becoming a core component of modern software systems. Companies today are integrating AI into customer support, analytics, automation, healthcare, finance, education, and enterprise applications. However, as organizations start building AI-powered solutions, one major question appears: “How do we make an AI model work specifically for our business needs?” Many teams immediately assume they need to train their own AI model. Others believe a well-written prompt is enough. Some organizations invest heavily in fine-tuning without understanding whether it is the right approach. The reality is that there is no single solution. Modern AI development usually revolves around three major strategies: Prompt Engineering Retrieval-Augmented Generation (RAG) Fine-Tuning Choosing the wrong approach can lead to higher costs, poor AI performance, security issues, and unnecessary complexity. This article explains the differences between these approaches and how businesses can select the right AI strategy. The Problem: Making General AI Models Business-Specific Large Language Models (LLMs) such as GPT, Claude, Gemini, and Llama are trained on massive amounts of publicly available data. They are excellent at: Understanding language Generating content Writing code Answering general questions Summarizing information ** However, businesses usually need AI systems that understand:** Internal company documents Customer information Product knowledge Industry-specific terminology Private databases Business processes For example: A hotel company wants an AI assistant that can answer: “What is our cancellation policy for premium customers?” A general AI model does not know this information because it was never trained on the company’s private policies. So the challenge becomes: How do we customize AI without rebuilding an entire model from scratch? This is where Prompt Engineering, RAG, and Fine-Tuning come

How do you actually test an AI agent? Not "does it respond," but: does it route to the right tool, chain calls correctly, recover from failure, resist prompt injection, and stay within cost/latency budget? I spent weeks working through this on a running agent, and open-sourced the entire methodology — framework-agnostic , so it applies regardless of your language, runtime, or toolset. What's inside •⁠ ⁠ 61-source benchmark map — BFCL, GAIA, τ-bench, SWE-bench, WebArena, AgentDojo, LongMemEval and more, categorized by what they actually measure •⁠ ⁠ 58 universal test blocks across 7 tiers (L1–L4, Error Recovery, Multi-Turn, Security). Each block = a tool-agnostic capability definition + a concrete reference implementation •⁠ ⁠ Full OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) mapped to 6 universal security test blocks •⁠ ⁠ Evaluation methodology — LLM-as-Judge biases, pass@k vs pass^k, trajectory vs end-state, observability (OpenTelemetry GenAI), automated red-teaming (garak, PyRIT, DeepTeam) •⁠ ⁠ Regulatory alignment — NIST AI RMF, MITRE ATLAS, EU AI Act, ISO/IEC 42001 How to use it Take Part II, replace the reference-implementation fields with your own agent's tool names and expected outputs. The universal capability definitions need no changes. Blank templates are included. PheronAgent (a macOS agent with 50+ native/MCP tools) is included as a real reference case study — but the methodology is the product, not the agent. No marketing narrative: ⁠ STORY.md ⁠ documents the real bugs, real test runs, and real corrections that shaped each version. Docs are CC BY 4.0, templates are MIT. Issues and PRs welcome. 👉 https://github.com/trgysvc/AgentTestMethodology