标签：#LLM

Book: Agents in Production — Building, Tracing, and Shipping Multi-Step AI You Can Trust Also by me: Observability for LLM Applications — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You ship an agent that resolves billing disputes. It works in the demo. Two weeks later a support ticket lands: the agent tried to refund $4,000 on a $19 charge. You read the trace. The model returned a JSON blob, your code did json.loads , pulled amount , and passed it straight to the payments API. No cap. No type. No check. The model hallucinated a number and your code trusted it. The model is stochastic. Your code does not have to be. The gap between those two facts is where most production agent bugs live, and it is exactly the gap Pydantic AI is built to close. The wedge is types Most agent frameworks hand you an Agent object and a bag of strings. Pydantic AI hands you Agent[Deps, Output] — a generic parameterized by its dependency type and its output type. The IDE and your type checker read those parameters. So does the runtime. Install pulls in the framework plus an optional tracing extra: pip install "pydantic-ai[logfire]" The smallest program that earns its keep: from dataclasses import dataclass from pydantic import BaseModel from pydantic_ai import Agent , RunContext @dataclass class Deps : customer_name : str class SupportReply ( BaseModel ): reply : str escalate : bool agent = Agent ( " anthropic:claude-opus-4-8 " , deps_type = Deps , output_type = SupportReply , system_prompt = " You are a support agent. " , ) A tool is a plain function whose type hints become the schema the model sees, and the run returns the validated SupportReply : @agent.tool def customer_name ( ctx : RunContext [ Deps ]) -> str : return ctx . deps . customer_name result = agent . run_sync ( " What is my name? " , deps = Deps ( customer_name = " Ana "

The Problem That Wouldn't Leave Me Alone Pakistan has 220 million people. A functioning legal system. Hundreds of Acts, ordinances, and constitutional provisions that technically protect every citizen. Almost nobody can use them. The median lawyer's consultation fee in Karachi is more than what many families earn in a week. Legal aid is understaffed and geographically concentrated in major cities. And the laws themselves? Written in English — a language most of the population reads functionally at best, and doesn't speak at home at all. So when a landlord illegally locks someone out. When a factory worker gets fired without severance. When a woman wants to know her inheritance rights. When a tenant needs to understand what "Section 16 of the Rent Restriction Ordinance" actually means for their specific situation — they either find a lawyer they can't afford, ask someone who doesn't really know, or quietly give up. This isn't a knowledge problem. It's an access problem. I'm a CS student at Sukkur IBA University in interior Sindh — not Karachi, not Islamabad. The kind of city where you feel the gap between what the law says and what people actually know it says every single day. That gap is where HAQ started. HAQ is an Arabic and Urdu word. It means right — as in, what is rightfully yours. The name felt important. The Core Idea: Ask the Law, Get the Law There's a specific failure mode with AI and legal questions that drove every design decision I made, and it's worth naming clearly. Standard LLMs — any of them — will answer legal questions confidently. They'll cite "Section 144" or "the Transfer of Property Act" with total authority. They are often wrong. Sometimes subtly: the section exists but doesn't say what the model claims. Sometimes obviously: the Act doesn't apply in that province. Always uncitable: the user has no way to verify without finding the source themselves. For an accessibility tool, a confidently wrong answer isn't neutral. It's actively dangerous.

Half a year ago, I wanted to see for myself what can we currently have with local LLMs. I went down the rabbit hole, learned quite a lot in the process, and shared my results in an article . The results were pretty discouraging: even with 32 GB VRAM, the best models I could run were both too slow and too dumb. At the same time, what you could get for free from inference providers was actually decent - and much faster. I remember my conclusion: "Let's wait for the next generation of models, which looks very promising. If we can run something comparable to full-size Qwen3-Coder-480B locally, that would be year of the Linux Desktop age of fully capable local LLMs. And now this day has arrived. Models Half a year later, I'm revisiting this question. And this time, the whole situation has turned upside-down. Almost none of the providers still have free tier, and anything that's still free is barely good enough even for the simplest tasks. And is rate-limited all over. And on the local side, the next Qwen lineup is out. So, that's what I'm going to be looking at. Once again, I have two RX6800's, 16 GB each, and 64 GB RAM. On one hand, this is more VRAM than any "normal person" can have with one GPU - unless you've got something specifically for AI, like an unified-memory Mac or a DGX Spark. On the other hand, RX6800 is "pre-AI" - anything newer will have much better performance thanks to tensor processors. Qwen3.6-27B : This is a dense model, so basically you can't run it at all on anything less than 32 GB VRAM. It's the slowest one, but also the best one if you can run it. Its accuracy is claimed to be on par with Claude 4.5 Opus, and better than Qwen3.5-397B-A17B . This is what I've been waiting for. It runs reasonably fast on my setup, so it's very much usable both in terms of performance and accuracy. Qwen3.6-35B-A3B : This one is MoE, and it's pretty small, so it's the fastest one. It's good for anything that doesn't require too much (i.e. for agentic tasks that don'

I've noticed something about myself. It's become much harder to get things done. I see less movement compared to my previous self. And sometimes, when I see people get so much done, tasks, projects, anything, I wonder why I can't make progress like that. Now I'm not comparing myself to Elon Musk here, I'm just wondering if I'm doing my best. For the longest time, I've wanted to build a portfolio website. But for some reason or another, I never actually ended up making it, or should I say, starting it. Whenever I had a free block of time, I wouldn't know where to start. I also have this side project I've been wanting to work on. I started it, then never made any more progress beyond the initial feasibility analysis. Why did I never allocate time to either? The Overthinking Starts Early I wanted the portfolio website to be really good. I wanted cool additions, beyond the usual "about me" and "projects" sections, I wanted things like "songs I'm obsessed with right now" and "last night I slept at." I was also looking if my fitbit has an API to show my live heart-rate on my portfolio (why would someone do that?). I was obsessed with the features, and with using the right tech to make it as efficient as possible. What stack should I even use for a portfolio? Same story with this very blog. Before writing a single word, I was wondering what stack to use. I looked into a ton of existing blogs for inspiration, how do I track views, what about comments, how do I stop a DDoS from flooding my DB if I'm storing comments, should I require auth just to leave a comment, should I support font and background color changes, themes, what's the best tool to do all of that? I've noticed this pattern with almost everything I do. I'm so obsessed with doing things right on the first attempt that I never actually make the first attempt, because "doing it right" is just too demanding. Or a related trap, when I want to do something perfectly , I discover prerequisites, and those prerequisites

You wire up the OpenAI SDK, ship the feature, and it works. Three weeks later someone in finance forwards a screenshot of a bill that tripled and asks what happened. You open the provider dashboard, see one big number, and… that's it. No per-feature breakdown, no idea which change caused it, no way to tell whether it's a bug or just growth. I've watched this happen at enough teams that I now treat "we can't explain our AI bill" as a predictable stage every company hits about two months after their first LLM feature ships. Here's how to get ahead of it — starting with plain code, then the tradeoffs, then where a dedicated tool actually earns its keep. Disclosure up front: I work on StackSpend, which does the full version of this. I've kept the first 80% of this post vendor-neutral because most of it you can and should build yourself before you buy anything. The core problem: the bill is a single number, your costs are not Provider dashboards give you total spend over time. What you actually need to make decisions is spend broken down by the dimensions you care about: Per feature — is it the summarizer or the chat assistant that's expensive? Per customer / tenant — which accounts cost more to serve than they pay? Per model — how much are you spending on GPT-4-class vs cheaper models? Per environment — is a runaway staging job quietly burning money? None of those dimensions exist in the raw bill. You have to attach them yourself, at call time, because after the request is gone the context is gone with it. Step 1: capture usage at the call site Every major provider returns token usage in the response. The trick is to log it with your own business context attached — the feature name, the tenant, the environment. Here's the pattern in TypeScript with the OpenAI SDK: import OpenAI from " openai " ; const openai = new OpenAI (); // Prices per 1M tokens — keep these in config, they change often. const PRICING : Record < string , { input : number ; output : number } > = { " g

When I published the first article about the Neuron AI Router , I expected questions about routing rules. Which rule to use for structured output, how to write a custom one, how the round robin behaves under load. Some of those questions arrived, but the most frequent one was different, and it wasn't really about routing at all. It was about failure. What happens to my agent when the provider goes down? It is a fair question, and if you are new to building AI applications it deserves a proper answer before we look at any code. Here is the short version. The new fallback strategy in Neuron AI Router lets you define an ordered list of LLM providers for your PHP agent. When an inference call fails with a transient error, such as a rate limit, a timeout, or an overloaded server, the same request is automatically retried on the next provider in the list. The failover is transparent: the agent never knows it happened, and the conversation continues without losing state. The rest of this article explains why this problem exists, why the usual solutions fall short, and how to configure it. Why LLM providers fail in production An LLM provider is an external service you talk to over HTTP. Every time your agent thinks, it is making a network call to a machine you don’t control, operated by a company that is currently serving millions of other requests. These services fail in very ordinary ways. You hit a rate limit because your traffic spiked. The provider returns an “overloaded” error because their traffic spiked. A request times out. A deployment on their side causes a few minutes of elevated error rates. None of this means you did something wrong, and none of it is rare. If you keep an agent in production long enough, you will see all of these. In a classic web application, a failed call to a third party API is usually a corner of the system. You log it, maybe retry it in a queue, and the rest of the page still works. In an agent based application the inference call is not

Spanlens is an open-source (MIT) LLM observability platform that lets developers monitor every call their application makes to OpenAI, Anthropic, Gemini, Mistral, OpenRouter, Azure OpenAI, or a local Ollama model. Integration takes one line: swap your client's baseURL to the Spanlens proxy, or run "npx @spanlens /cli init" and the wizard rewrites your code automatically. From that moment, every request is recorded with its model, token counts, latency, cost, and full prompt and response body, with streaming responses reconstructed automatically. The dashboard turns that raw log into operational insight. Cost tracking breaks spend down per request, per model, and per end user, and parses prompt-cache tokens separately so you see real cache savings rather than sticker price. Agent tracing visualizes multi-step workflows as Gantt waterfalls and node-and-edge graphs, highlighting the critical path so you can find the slowest dependency chain in a fan-out. Anomaly detection flags 3-sigma deviations in latency, cost, or error rate against a rolling 7-day baseline with root-cause hints. Alerts on budget, error rate, and p95 latency are delivered to Email, Slack, or Discord. Spanlens goes beyond passive logging. A regex-based PII and prompt-injection scanner inspects request and response bodies and can block injections at the proxy. The savings engine spots calls that match a cheaper model's profile (for example, a gpt-4o call that looks like a classification task) and estimates the monthly saving from switching. Prompt versioning with A/B experiments compares versions on latency, cost, and error rate using Welch's t-test for statistical significance, and an LLM-as-judge evaluation framework (judge with OpenAI, Anthropic, or Gemini) scores outputs against rubric anchors, with human agreement measured by Pearson r or Cohen's kappa. Reusable datasets power offline evals and regression checks.

At the beginning of June I started a “ Claude withdrawal ” challenge. The plan was to run MiniMax 3 for a month, to see if I can get the same level of quality, but at 5x less the price. Until then, Claude Code was my main driver, with MiniMax on the backup, for when I was running out of quota, or sometimes for code review. The monthly bill for Claude was $100 on the Max plan, whereas for MiniMax I would pay $20 for the Token plan. All in all, it seemed like an interesting experiment. Then, half way through the challenge, Grok came into the picture. I got a very interesting offer at $35 for 3 months, then $35/month. But Grok has something neither Claude, nor MiniMax can give me out of the shelf: video and image generations. The only unknown was if switching from Claude Code to Grok will still maintain the same coding power. So I instantly took the offer, and did whatever I had to do to understand if this was the right path. And here comes the “whatever I had to do”, in plain technical terms. Switching from Claude Code to Grok – the Actual Steps The switch itself was interesting because I didn’t want to lose the Claude Code interface. I like the harness. The way it works with my codebase, the commands, the flow. So I used a helper called cliproxyapi . It’s a small proxy that sits between the Claude Code client and whatever model you point it at. You run it locally, tell it to forward requests to Grok’s API instead of Anthropic’s. Then you launch Claude Code the same way you always do, but it talks to Grok under the hood. Here’s how it goes in practice. Step 1: Install the proxy. I used brew to install it, I’m on a Mac, and also because I wanted to have it started as a service. Step 2: Set two environment variables. One is the target API base URL, for Grok that’s something like https://api.x.ai . The other is your API key. "env" : { "ANTHROPIC_BASE_URL" : "http://localhost:8317" , "ANTHROPIC_API_KEY" : "cliproxy-local-key" } , Notice how we use “cliproxy-local-key”, be

Structured output broke on us three times. The third time taught us what "operator-ready" means. Last quarter we shipped a contract-extraction agent to an enterprise legal team. Schema validation passing at 97%. Human reviewers satisfied with the output quality in testing. Rollout went smoothly. Then it broke. Three times. In three completely different ways. The first two failures we fixed with better prompts and stricter schemas. The third one taught us something the first two hadn't: that "operator-ready" is not a technical checklist. It's a claim about your agent's behavior under conditions you didn't design it for. Failure one: the validation paradox Week two. A lease agreement came through with a renewal clause formatted as a table instead of prose. Our extractor looked for renewal terms in a specific JSON path. The table format populated the schema differently. Validation passed. The extracted renewal date was off by two years. The fix was obvious in retrospect: add a canonical-format normalization step before extraction. But the lesson was sharper than that. Schema validation tells you the shape of the output, not whether the content is correct. A JSON object with the right keys and the right types can still contain wrong values. Our 97% validation success rate was measuring the wrong thing. It was measuring structure conformance, not content accuracy. After this failure, we separated validation into two signals: schema validity (does the object have the required fields) and field confidence (do we have evidence the content is correct). We started logging both. An output is trusted only when both signals are above threshold. Failure two: the retry loop that lies Month one. A particular clause type appeared in a contract format we hadn't trained our test set on. The extractor failed schema validation on the first attempt. Our retry logic kicked in, filled missing fields with model-inferred defaults, and passed validation on the third try. The output looked rig

You built a system to surface information so a person could decide. Somewhere it started deciding for them — the output stopped saying "here's what the documents show" and started saying "you should do X." Nobody designed that drift. An LLM, when asked a question, produces an answer-shaped thing, and an answer easily becomes a verdict. What everyone tries A prompt instruction: "Don't make recommendations." "Only state what's in the documents." People add the line and assume the boundary is enforced. Why it doesn't work A prompt instruction is a request, not a guardrail. The model follows it most of the time, then on the input that matters produces a confident recommendation anyway, because nothing structurally prevents it. "Please don't make recommendations" is to a guardrail what a sticky note saying "please don't enter" is to a locked door. And the stakes are higher than they look. When output drifts from evidence to verdict, accountability moves. As long as the system returns evidence and a human decides, the human owns the decision. The moment the system returns a verdict and the human defers, the system is deciding things it was never validated to decide — and when one is wrong, accountability is a blank. High-stakes fields separate evidence extraction from judgment on purpose; most RAG systems erase that line by default. The one shift Decide what the output is and enforce it structurally. An output should declare itself: answer, evidence, missing facts, or out-of-scope. "Return decision material, not a decision" has to live in the output contract and in gates — not in a polite request to the model. The system supplies frames; the human supplies verdicts. This is the output boundary — one of three places production RAG dies. Read the full version on my blog , where this connects to the RAG Failure Diagnosis Kit for teams debugging production RAG.

Most backend engineers already know how to handle HTTP errors. 400 means the request is bad. 401 means auth failed. 429 means rate limited. 500 means something broke upstream. Retry a few times, add exponential backoff, log the response body, move on. That works fine for many APIs. It works badly for LLM APIs. LLM providers may use normal HTTP status codes, but the operational meaning behind those errors is different enough that treating them like ordinary REST failures can make your app slower, more expensive, and harder to debug. The mistake I kept making Early on, I handled LLM failures the same way I handled every other external API: if ( response . status === 429 || response . status >= 500 ) { retryWithBackoff (); } Simple. Familiar. Dangerous. That logic misses the actual question your app needs to answer: What kind of LLM failure happened, and what should the product do next? Because an LLM API failure is rarely just "one HTTP request failed." It can break: a user-facing chat response a background agent run a document generation job a tool-calling workflow a batch evaluation pipeline a structured JSON generation step And each one needs different handling. Not all 429s mean the same thing For a normal API, 429 Too Many Requests usually means: Slow down and retry later. With LLM APIs, 429 can mean several different things. It might be a temporary rate limit: { "error" : { "message" : "Rate limit reached" , "type" : "rate_limit_error" } } Retrying with backoff may help here. But it might also mean quota exhaustion: { "error" : { "message" : "You exceeded your current quota" , "type" : "insufficient_quota" } } Retrying this does not help. It just adds latency, noisy logs, and a worse user experience. It could also be model-specific pressure. One model may be overloaded while another model from the same provider, or a different provider, would work fine. So your handler should distinguish between: temporary rate limit hard quota exhaustion model-level capacity is

Yapay zeka mühendisi (AI Engineer) olmak, yalnızca ChatGPT’ye veya Claude'a akıllıca promptlar yazmaktan ibaret değildir. Yapay zeka modellerini kullanarak gerçek dünyadaki karmaşık problemleri çözen, sürdürülebilir, güvenli ve ölçeklenebilir yazılımlar inşa etmek ciddi bir mühendislik disiplini gerektirir. Eğer temel basamakları sağlam kurmadan doğrudan en üstteki otonom ajan (agentic) yapılarına atlamaya çalışırsanız, üretim ortamına çıktığınızda beklenmedik hatalarla (hallucination), kontrol edilemeyen maliyetlerle ve aşırı yüksek yanıt gecikmeleriyle (latency) karşılaşırsınız. Bu rehberde, sıfırdan başlayıp üretim seviyesine (Production-ready) kadar uzanan 17 katmanlı AI Engineer Yol Haritası'nı adım adım inceleyeceğiz. Phase 1: Foundation (Temel Katman) Binalar gibi, yapay zeka uygulamaları da güçlü temeller üzerine kurulur. Bu aşamada amacımız yazılım geliştirme ve temel model etkileşim mekanizmalarını kavramaktır. Python & Data Yolculuğun ilk ve en kritik basamağı Python programlama dili ve veri yönetimidir. Yapay zeka dünyasında veri okuma, temizleme, dönüştürme ve analiz etme süreçleri günlük işlerin büyük kısmını oluşturur. Python'ın liste, sözlük gibi yerleşik veri yapılarını, nesne tabanlı programlama (OOP) mantığını ve fonksiyonel yapısını kavramak şarttır. Ayrıca veri manipülasyonu için Pandas ve sayısal işlemler için NumPy gibi kütüphanelerde yetkinlik kazanılmalıdır. Detaylı bilgi için Python Resmi Dokümantasyonu incelenebilir. Software Engineering & APIs Kodunuzun lokal bilgisayarınızda sadece "çalışması" yeterli değildir. Üretim ortamında çalışacak kodun temiz, okunabilir, sürdürülebilir ve test edilebilir (Unit/Integration Tests) olması gerekir. Ayrıca modelleri uygulamalara entegre ederken REST API tasarımları, authentication (kimlik doğrulama) mekanizmaları, asenkron programlama, hata yönetimi (Error Handling) ve servis mimarileri hayati önem taşır. Git gibi versiyon kontrol sistemleri ve temel Docker bilgisi bu katmanın ayrılmaz parçasıdır. Pro

Visibility Collapse in the Post-LLM Engineering Stack Artificial intelligence is still improving. But something important has changed in how that improvement is perceived. For developers and engineers working closely with frontier models, the experience is no longer one of explosive capability jumps. Instead, it feels like: incremental improvement under increasing structural constraints This shift is not about stagnation. It is about uncertainty in how AI capability is exposed, deployed, and interpreted. Capability vs Visibility: the new separation Recent frontier model systems (such as Fable 5, as described in industry discussions) highlight an important architectural pattern: Certain capabilities are no longer fully exposed in production environments: advanced coding assistance deep debugging autonomy bioinformatics reasoning cybersecurity-related reasoning This does not necessarily imply reduced model capability. Instead, it reflects a system-level separation: model capability ≠ deployed capability System interpretation: Modern AI stacks are becoming layered systems: Raw Model → Safety Layer → Policy Filter → Deployment Interface → User Access This means developers are no longer interacting with models directly. They are interacting with constrained capability surfaces. Perceived slowdown in LLM progress Despite continued benchmark improvements: reasoning scores increase gradually multimodal capabilities expand tool-use frameworks improve The perceived acceleration of AI has weakened. Compared to 2022–2023, there are fewer qualitative jumps. From an engineering perspective, this suggests a transition: from capability discontinuity → capability smoothing In other words: AI is still improving, but improvements are less visible at the system interaction level. Economic mismatch: scaling vs returns The AI ecosystem is currently defined by a structural tension: Inputs: massive GPU infrastructure investment multi-billion-dollar training runs hyperscaler-scale capital a

A large-scale audit of AI-as-judge evaluation — covering over half a million individual judgments — finds that AI judges are consistently reliable but not valid, meaning they give the same answer repeatedly without that answer being correct. Published work and popular benchmarks like Chatbot Arena have treated consistency as proof of trustworthiness, and the audit shows that assumption is unfounded. Key facts What: Using one AI to grade another is now common — but the biggest audit yet shows these graders are consistent without being correct. A judge that always picks "answer A" scores perfectly on consistency. When: 2026-06-19 Primary source: read the source (arXiv 2606.19544) The distinction matters: a judge is reliable if it's consistent (same question, same answer), and valid if those answers are actually correct. The audit's central finding is that AI judges are reliable without being valid, and the field has been treating the first as evidence of the second. Because consistency is easy to measure and looks reassuring, it has stood in for actual trustworthiness across a lot of published work. A new audit makes the problem stark: a judge that ignores both answers and always picks the one labeled "A" would be perfectly consistent — flawless reliability, identical verdict every time — and completely worthless, because it never read anything. Consistency is trivially easy to fake and says almost nothing about whether the judging is sound. Yet "the judge agrees with itself" has done significant reassurance work in papers and benchmarks, and the always-pick-A example shows exactly how empty that reassurance is. When the researchers corrected for the agreement you'd get by chance — as any fair test should — confident-looking scores deflated noticeably. Gaps between models that seemed meaningful shrank or blurred. Accepted folk wisdom also took a hit: the long-standing worry that AI judges are suckers for longer, wordier answers turned out to be far weaker than assumed

Key Takeaways You can't unit-test a coach agent the way you test a pure function — the output is non-deterministic and "good" is a judgment call, not an assertion. An LLM-as-judge harness lets you grade a whole test set automatically against a rubric, which is the only way solo-scale eval stays sustainable. But the judge is itself a fallible model. If you don't design around its known biases — position, verbosity, self-preference, and quiet drift when the judge model updates — you build a green dashboard that means nothing. The mitigations that actually work are mechanical, not prompt-magic: shuffle order on every pairwise call, pin the judge version, keep a small human-labelled anchor set, and re-check the judge against it. The problem I actually had FamNest's coach agent generates responses to parents — check-ins, encouragement, the occasional gentle redirect. I have a growing pile of these interactions, and every time I change a prompt, swap a model, or adjust the pipeline, I need to know one thing: did I just make it better or worse? For normal code, that's what tests are for. I change something, the suite runs, red or green, done. But there's no assertEqual for "was this an empathetic, useful response to a tired parent." The output changes every run even at temperature zero-ish, and the quality bar is a human judgment, not a fixed string. Two responses can be worded completely differently and both be good. One can match my "expected output" word for word and still be worse than a version that didn't. So the honest options were: read every response by hand every time I change something (does not scale past about week two), or build a harness where a model grades the outputs against a rubric. I built the harness. Then I spent an uncomfortable amount of time learning all the ways a harness like that can lie to you. What the harness actually is At its simplest, it's a loop: def evaluate ( test_cases , coach_agent , judge ): results = [] for case in test_cases : res

A ChatGPT subscription starts at $20 a month and is one of the cheapest ways to run inference. OpenAI has also been fairly relaxed lately about third-party agents using them , which makes the deal even better for a lot of us. But a subscription can't be used as freely as pay-per-token access , and the providers police the difference. Anthropic recently narrowed its subscriptions to first-party apps; OpenAI has its own limits. Here's what will get you banned from an OpenAI subscription. Sharing your subscription A ChatGPT subscription is strictly personal. One subscription, one user. Sharing yours breaks OpenAI's terms of service. That also covers account pooling and account rotation, where several people share the same credentials to dodge rate limits. Running it in automation Automation (CI, runners, schedulers) should run on per-token pricing, not a subscription . Once a system calls the OpenAI API with your token while you're not in the loop, the usage stops being personal. No unattended production system should run on a ChatGPT subscription. Serving other users For now, you can point an autonomous agent like OpenClaw or Hermes at your ChatGPT subscription, as long as it only talks to you. The moment that agent starts chatting with other people, or serving them in any way, it turns into a team use case , and that inference should be paid per usage. Putting it in a commercial product Same logic here. Making an LLM call authenticated with an individual ChatGPT subscription inside a product you ship breaks OpenAI's terms. That access is subsidized, and reselling it in any form isn't what it's meant for. If you've built something just for yourself and you're the only user, you're probably fine. The bottom line A ChatGPT subscription is personal . Anything that stretches past personal use can get you restricted or banned. If you're not sure your usage counts, move it to pay-as-you-go. If you want to keep the subscription for your own work and fall back to per-token pr

Tree of Thoughts was a genuine leap. Instead of reasoning in one straight line, it branches into several lines, scores them, prunes the dead ends, and searches for the best path — so a puzzle that would sink a single chain of thought becomes solvable. But a tree has one restriction baked right into its shape, and once you see it you can't unsee it: every node has exactly one parent. A branch can be extended or abandoned. It can never be combined with another branch. That matters more than it sounds. Real problems decompose, and when they do, different branches each get part of the answer right. Branch A nails the first half; branch B nails the second half; neither is fully correct on its own. A tree is forced to pick one and throw the other's good half away. Graph of Thoughts (GoT) removes exactly that restriction. 🕸️ Interactive demo (a real merge-sort that branches, merges, and refines — with live-verified scores): https://dev48v.infy.uk/prompt/day21-graph-of-thoughts.html The core idea: thoughts are nodes in a graph GoT reframes reasoning as building a graph . Each vertex is a thought — a partial solution or intermediate result. Each edge is an operation that produced one thought from one or more others. Because it's a general graph and not a tree, a thought is allowed to have several parents, and edges can even loop back on themselves. That single change in the data structure is the entire conceptual leap. Everything else is just the operations you're now free to run on that network. The four operations generate (branch) — the familiar move, straight from Tree of Thoughts. From one thought, produce several different next thoughts. This can also be a split : break the input into independent sub-problems solved on separate branches. Diversity matters here — near-duplicates waste budget. score / rank — turn each thought into a number so the controller can compare them. Objective scorers win: a validator, a test, a metric. In the demo, the scorer is deliberately con

Contorium is a local-first system that introduces persistent project cognition into AI-assisted development workflows. Instead of treating AI as a tool that operates on code, Contorium treats the project itself as a structured, evolving system. ⸻ 🧠 Problem Modern AI coding workflows suffer from a structural limitation: Even with tools like: Cursor Claude Code MCP-based agents IDE copilots context is still: fragmented session-based non-persistent weakly structured This leads to: repeated explanations, lost reasoning, and architectural drift ⸻ 🧩 Solution: Project Cognitive Runtime (PCR) Contorium introduces a runtime model where project understanding is persistent and structured. ⸻ Core Components ⸻ PIK — Project Intent Kernel PIK defines the system-level intent of a project: primary goal constraints non-goals priority weighting It acts as a stable semantic anchor. ⸻ CIL — Cognitive Interaction Layer CIL captures reasoning: why decisions were made what alternatives were considered how context influenced outcomes It makes reasoning persistent instead of ephemeral. ⸻ Timeline Layer All system changes are recorded as events: code changes AI outputs tool interactions architectural decisions This enables replay and evolution tracking. ⸻ Drift Detection Layer A continuous alignment system compares: current behavior vs PIK intent It detects: intent drift structural drift behavioral drift And produces measurable deviation signals. ⸻ 🔁 System Loop Contorium forms a continuous loop: PIK defines intent Execution produces behavior Timeline records evolution Drift system evaluates alignment Suggestions guide correction This creates a self-regulating project system. ⸻ 🧠 Key Insight Contorium is not an AI coding tool. It is a: Project Cognitive Runtime (PCR) A system where software projects maintain structured intelligence over time. ⸻ 🚀 Why it matters The bottleneck in AI development is no longer capability. It is continuity of understanding across: time tools agents sessions Conto

If you run a RAG system in production, you eventually hit a dilemma that has nothing to do with your model and everything to do with your cache. Cache the answers to save tokens and latency, and one day a source document changes — but your cache keeps cheerfully serving the answer it built from the old document. Nobody gets an error. The number is just quietly wrong. Cache nothing , and every single call re-retrieves the same chunks, re-reads them, and re-pays the full context bill to rebuild an understanding you already built five minutes ago for a nearly identical question. Stale or expensive. Most teams pick "expensive" because at least it's correct, then bolt on a TTL and hope. This post is about why the TTL doesn't save you, and about two specific, mechanical fixes that let you cache RAG context and stay fresh. I maintain an open-source library called Coalent that implements both, so I'll use it for the runnable examples — but the two ideas are portable and worth stealing even if you never pip install anything. Failure mode 1: the stale RAG cache (and why a TTL won't save you) Here's the standard "answer cache" sitting in front of retrieval: answer = cache . get ( query ) if answer is None : chunks = retriever . retrieve ( query ) answer = llm . synthesize ( query , chunks ) cache . set ( query , answer , ttl = 3600 ) return answer This works until billing.md changes. The refund window goes from 30 days to 14. Your cache has an answer keyed on "what is our refund policy?" that says 30, and it will keep saying 30 for up to an hour — or forever, if the same question keeps refreshing a TTL that never expires under load. The reason this is hard is that the cache key (the query) has no relationship to the thing that changed (the source). You cached an answer; you threw away the fact that this particular answer was derived from billing.md . So when billing.md changes, you have no way to find the answers that depended on it. The TTL is a confession that you can't answ

An AI feature can feel impressive and still be a bad product decision. The demo is fast. The answer sounds useful. The team is excited. Then usage grows and nobody can answer the basic questions: Is it accurate enough? Is it saving time? Which customers trust it? Why did costs spike? Should we scale it, fix it, or kill it? That is the trap an AI metrics baseline prevents. A baseline is not a dashboard full of vanity charts. It is a small set of before-and-after measurements that tells you whether an AI workflow is getting better, getting worse, or merely getting more expensive. Why AI features fail without a baseline Most software teams already track uptime, errors, and conversion. AI features need those too, but they also need new signals because model behavior is probabilistic. A normal API either returns the expected response or throws an error. An AI workflow can return: a fluent answer that is wrong a correct answer with missing evidence a useful answer that costs too much a slow answer that users abandon a safe answer that refuses too often a cheap answer that hurts trust a high-rated answer that does not improve the business workflow Without a baseline, every production discussion becomes opinion-driven: "The model seems better." "Users like it." "The new prompt reduced hallucinations." "The expensive model is worth it." Maybe. Maybe not. The baseline turns those claims into measurable comparisons. What an AI metrics baseline is An AI metrics baseline is the starting measurement for the workflow before you optimize or scale it. It answers five questions: What does the workflow cost today? How good are the outputs today? How fast and reliable is the experience today? Do users adopt and reuse it? Does it improve the real task it claims to improve? You do not need 80 metrics on day one. You need a small set of metrics that match the feature's risk and purpose. For example: Feature Useful baseline Support answer bot resolution rate, citation quality, escalation r