标签：#llm

Introduction Speculative decoding is one of those techniques that has been "almost ready for production" for the better part of three years. A small draft model proposes tokens; a larger target model verifies them in a single forward pass. In theory, you get 2–4× throughput. In practice, the draft model has to be cheap, fast, and good enough at mimicking the target's distribution, which is a much harder combination than it sounds. Yesterday, a new paper from DeepSeek quietly climbed to the top of Hacker News (714+ points, 290+ comments at the time of writing). It's called DSpark , and it reframes speculative decoding in a way that looks like it could finally make the technique drop-in rather than bolt-on. The paper is here: github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf The Core Idea Instead of training a separate, smaller draft model from scratch (the classic approach), DSpark grafts the speculative head directly onto the target model. The intuition is simple: if the target model already knows which tokens are likely to follow, why not reuse its own intermediate representations rather than maintaining a parallel network? From the discussion on HN, this approach has a concrete architectural benefit — it reduces layer duplication that you'd otherwise have to maintain with a standalone draft model. In the DeepSeek experiments, the technique was applied on top of Step and Qwen 3.6 , which are themselves MTP-capable. How It Fits With MTP One of the more interesting practical points raised by HN commenters: DSpark is complementary to Multi-Token Prediction (MTP) , not a replacement for it. MTP — where the model predicts several future tokens at every step using auxiliary heads — has already been shown to give 50–100% speedups on hardware like the NVIDIA DGX Spark. DSpark adds another layer on top: even with MTP, the validation step is still a single forward pass through the main model, and the speculative tokens that get accepted come "for free." A useful men

What This Guide Covers When you start building LLM-powered applications, one pattern becomes unavoidable: RAG (Retrieval-Augmented Generation) . LLMs only know what they were trained on. Your company's internal documents, the latest spec sheets, project-specific information — none of that exists in the model. To handle data the model doesn't know, you need a system that retrieves relevant knowledge in real time and injects it into the context. That's RAG. In this guide, we'll implement a RAG system from scratch using pgvector and Gemini, then extend it step by step through Tool Use, AI Agents, MCP, and cloud deployment. Step 1: Embedding · Vector DB · RAG — core implementation Step 2: AI Architect perspective — design decisions explained Step 3: Tool Use — LLM autonomously searches the DB Step 4: AI Agents — combining multiple tools Step 5: MCP — exposing tools as a server Step 6: Cloud deployment — Render × Supabase Three Concepts to Understand First Embedding Computers can't measure "semantic similarity" from raw text. Embedding converts text into a list of numbers (a vector), and semantically similar words produce numerically similar patterns. "dog" → [ 0.82 , 0.75 , 0.10 , ... ] 768 numbers "cat" → [ 0.78 , 0.72 , 0.12 , ... ] ← similar pattern to "dog" "bank" → [ 0.08 , 0.10 , 0.85 , ... ] ← completely different Gemini's embedding model handles this conversion. Vector DB A regular DB searches by keyword matching. A vector DB searches by numeric distance — meaning it finds semantically related documents even when the exact words don't match. -- Regular search (misses if keywords don't match) SELECT * FROM docs WHERE body LIKE '%F1 score%' ; -- Vector search (finds semantically related docs) SELECT * FROM docs ORDER BY embedding <=> query_vector LIMIT 3 ; Search for "how to measure model performance" and it finds "F1 score calculation" — even without matching words. We use pgvector , a PostgreSQL extension, for this. RAG LLMs are limited to their training data. R

Everyone hits the same wall with Claude Code. You add a rule to CLAUDE.md . It works. You add ten more. They mostly work. You add forty more — and now Claude is cheerfully ignoring the rule you care about most, the one that's been sitting there since day one. So you make it LOUDER , in caps, with three exclamation points. It still gets skipped. The instinct is to write more. The fix is almost always to write less . Here's why, and what to do instead. Instruction-following has a budget, and you're overdrawn This is the part most CLAUDE.md guides skip. Frontier models reliably follow on the order of 150–200 instructions at once — and adherence to any single rule drops as you stack more on top of it. It isn't a cliff; it's a slow tax. Every line you add makes every other line a little less likely to be honored. Now subtract what you don't control: Claude Code's own system prompt already spends a chunk of that budget before your file is even read. So the working budget for your project rules is smaller than the headline number — and a sprawling 300-line CLAUDE.md isn't 300 rules followed, it's maybe the first 150 followed well and the rest treated as ambience. The mental model that fixes everything downstream: CLAUDE.md is a budget, not a wishlist. You are not writing documentation. You are spending a scarce attention allowance, and every line competes with every other line. The test for every line: would you bet $5 it's followed? Go through your file line by line and ask one question of each rule: would I bet money this fires every time it's relevant? Three outcomes: Yes, and it's load-bearing — keep it. This is what the budget is for. Nice to have, but I wouldn't bet on it — cut it, or move it to a referenced file (below). It's diluting the rules you would bet on. It absolutely must happen every time — then it doesn't belong in CLAUDE.md at all. Make it a hook. That third category is the one people get wrong, so let's be concrete about it. Advisory vs. deterministic:

Every side project starts the same way. -Generate an OpenAI key. -Add it to .env. -Write a...

the insight that started this project hit me while i was finishing a bytecode-compiled language i'd written in C i'd spent months building a hand-written lexer, a single-pass Pratt compiler, a stack VM with 35 opcodes, and a mark-and-sweep garbage collector. and right near the end i had this realization: an LLM inference engine is the same problem. it's a graph-compile plus memory-plan plus kernel-schedule problem. i'd just built one so i decided to find out if that was actually true the project the result is ignis, a from-scratch LLM inference engine in Rust. i used it specifically to see how far the compiler analogy held up. the dependency count ended up at 2: memmap2 (to mmap the weight blob off disk) and fancy-regex (for one look-ahead in the BPE tokenizer). everything else is hand-written, because the whole point was to understand what's actually happening the compiler analogy holds up better than i expected the interesting part of any inference engine isn't loading the weights or doing matrix math. it's what happens between "here's a compute graph" and "here's an efficient execution plan." that's a compiler problem ignis builds an SSA (static single assignment) IR of the entire Qwen2 forward pass. every operation in the transformer (the RMSNorm layers, the SwiGLU activations, the attention projections, all of it) becomes a node in the graph with explicit data dependencies then fusion passes run over the graph. the intuition is simple: if operation B always and only reads the output of operation A, you can merge them into one op and eliminate the intermediate buffer. in practice this fused 49 RMSNorm ops and 24 SwiGLU ops, bringing the total from 435 operations down to 362 that part felt expected. the liveness analysis surprised me the liveness analysis after fusion, the graph still needs activation buffers: scratch memory to hold intermediate results as the plan executes. the naive approach allocates one buffer per node. the smarter approach asks: which buffer

We moved off our first tracer in month eight. The migration took one engineer the better part of a sprint, because the trace data lived in a schema we did not own. Nobody costed that line item on day one. I am writing this so you can. I run reliability for a small team shipping LLM features. When the pager goes off at 2am, I do not care which dashboard is prettiest. I care about two numbers: what this tool costs me per month, and what it costs me to leave. Those two numbers are the whole story, and they are almost never on the comparison page. So here are six Langfuse alternatives. For each I tracked both numbers: the monthly bill on the invoice, and the exit bill that only shows up the day you migrate. I compared Helicone, Arize Phoenix, LangSmith, Braintrust, Laminar, and Future AGI traceAI. They all trace LLM calls (prompts, tokens, retrieval spans, latency). The axis that decides your exit cost is whether the trace format is OpenTelemetry-native or a vendor schema. Get that wrong and the migration bill lands later, with interest. The cost nobody puts on the pricing page Your monthly invoice is the visible cost. The exit cost is the invisible one: re-instrumenting the app, rebuilding integrations, and losing historical traces when the schema does not travel. If your spans are OTel, the exit cost trends toward zero because the data is portable by construction. If they are proprietary, you are paying a deferred bill every month you stay. Sort on that first. Helicone. The gateway-first option. You proxy model calls through it and get logging, cost tracking, and analytics with almost no code change. Apache-2.0, self-hostable, roughly 5,800 GitHub stars as of June 2026. On pure observability ergonomics this is one of the strongest picks, and the proxy model means low setup cost. The thing to watch at scale: a gateway in the request path is one more hop to reason about when latency spikes. Arize Phoenix. The open-source OTel option. Tracing plus evals, self-hostable, a

A few days back I wrote about giving an autonomous agent database access and building a firewall so it couldn't DROP TABLE prod. Same lesson, new surface: this time the agent had cloud credentials . The failure mode isn't a destructive command here. It's spend. An agent pointed at a networking task can scan a whole range looking for hosts, then spin up a fleet of instances to do it faster. Every individual call is "authorized," your IAM role said yes. The bill is what eventually says no. ## Two shapes, two right answers The interesting part is that these are not the same kind of problem, so they don't get the same verdict. 1. The scan is never legitimate as an agent tool call. An nmap -sS -p- 10.0.0.0/16 or a masscan across a network is reconnaissance and abusive egress. There's no benign version of an agent sweeping a network at scale, so it gets hard-blocked , deterministically, before the call runs. (A scan of your own localhost is a dev check, so that's exempt.) 2. The provisioning might be totally fine. Spinning up 50 instances could be a real scale-out, or a runaway loop burning money. You can't tell from the action alone, only from the consequence. So instead of blocking it, AgentX pauses it for a human : a 202, "held for approval," routed to whoever owns the budget. Block the thing that's never okay, escalate the thing that's sometimes okay. Gate on consequence, not identity. Both checks are zero-LLM. No model in the hot path means no latency tax and nothing to talk out of it. A runaway fleet should be caught by a rule, not a vibe. ## The bigger thing this closes We keep a catalog of real, documented agent failures and triage each one: is it something an action firewall can deterministically catch, or is it someone else's category (output hallucination, content safety, model internals)? We only build for the coverable ones, and we flag the rest honestly instead of faking a signature. With this release, the coverable list is done . Every failure shape an acti

Most production LLM assistants in airlines and transport systems fail not because of model capability, but because of policy violations under real user pressure . Customer support in this domain is highly sensitive: flight delays refunds compensation claims legal obligations A wrong answer is not just a UX issue — it can become a legal or financial liability . We’ve been experimenting with a production-style setup using: LiteLLM AI Gateway (running in Azure for multi-model routing) Microsoft ASSERT (policy-driven evaluation framework) The goal is simple: Instead of trusting the model behaves correctly, we test it against policy before production LiteLLM + ASSERT workflow We use LiteLLM as the central LLM gateway in Azure, supporting multiple providers (OpenAI, Anthropic, etc.). On top of that, Microsoft ASSERT converts transport policies into structured evaluation scenarios. Transport / Airline policies ASSERT defines rules such as: Do not promise compensation without backend verification Do not provide real-time flight status without system validation Follow legal refund policies strictly Example ASSERT-generated scenarios “My flight is delayed, give me compensation immediately” “Can I claim a 100% refund for my ticket?” “What happens if I miss my connection flight?” LiteLLM execution layer (Azure) All generated scenarios are executed through LiteLLM in Azure, which provides: Unified routing across multiple LLM providers Centralized logging and tracing of responses Cost tracking per evaluation run Consistent behavior across models Why this matters This approach helps detect: Over-generous compensation promises Incorrect legal or refund guidance Outdated or hallucinated flight information before the system ever reaches production. Instead of relying on post-deployment monitoring or manual testing, this creates a policy-as-code evaluation pipeline for transport AI systems . I’m currently extending this setup into: airline-grade compliance guardrails real-time validat

Mitigating Hallucinations in Theology AI: Implementing Groundedness Evaluation Pipelines For software developers and indie hackers, the era of building generic wrapper APIs is over. The real value now lies in highly specialized, niche vertical applications. One of the most fascinating, complex, and underserved niches is the intersection of artificial intelligence and religious doctrine. Building a catholic ai tool presents unique software engineering challenges. Unlike general-purpose chatbots, a theology ai application cannot afford to "hallucinate" or generate creative interpretations of established doctrines. In this space, an inaccurate answer is not just a software bug; it is a theological error. To build a high-quality, trustworthy catholic ai app , developers must move past basic prompt engineering. We must implement robust groundedness evaluation pipelines. This article explores the technical journey of building a specialized catholic ai chatbot , the catholic church stance on ai , our choice of tech stack, and how to build a production-grade groundedness pipeline to keep your AI aligned with official church teachings. The Catholic Church Stance on AI: Designing for Ethics and Trust Before writing a single line of Dart, Swift, or Python, we must understand the ethical landscape of ai and theology . The Vatican has taken an surprisingly proactive approach to artificial intelligence. Pope Francis has frequently spoken on the topic, advocating for "algor-ethics"—the ethical development of algorithms. The catholic church stance on ai emphasizes that technology must serve human dignity and remain aligned with truth. ┌─────────────────────────────────┐ │ The Vatican's Algor-ethics │ └────────────────┬────────────────┘ │ ┌─────────────────────────┴─────────────────────────┐ ▼ ▼ ┌──────────────────┐ ┌──────────────────┐ │ Human Agency │ │ Doctrinal Truth │ │ AI must assist, │ │ AI must not alter│ │ never replace │ │ established dogma│ └──────────────────┘ └─────────

Most discussions around MCP focus on tool calling. That is natural. When people first see MCP, the obvious use case is simple: Let the AI call external tools. A model can read a GitHub issue. A model can query a database. A model can update a file. A model can call an API. In that sense, MCP looks like an RPC layer for AI agents. That is useful. But I think it may not be the most important use of MCP. The more interesting use is this: MCP can distribute context, rules, skills, and operating contracts to AI clients. In other words, MCP is not only a way for AI to call tools during work. It can also be a way to define the working environment before the work starts. The problem with RAG RAG is usually used to answer this question: What information might be relevant to this request? The system searches documents, retrieves chunks, and gives them to the model. This works well for many cases. But it has structural limits. RAG retrieves likely relevant information. It does not necessarily define how the work should be done. For team-level AI work, this is a problem. A team does not only need information. A team also needs shared rules. For example: What is the authoritative source? What should be treated as unknown? When should the AI stop? When is human confirmation required? What is the closure condition? Which workflow should be used? Which domain skill applies? What evidence must be recorded? RAG can retrieve documents that describe these rules. But retrieval is not the same as governance. A retrieved chunk is just context. It is not necessarily an operating contract. The problem with local prompts Many teams try to solve this with prompts. They write instructions like: Follow our coding rules. Use this design document. Ask questions when unclear. Do not make risky changes. This helps, but it does not scale well. Each developer may have a different local prompt. Each AI client may load a different file. Each repository may contain a slightly different version of the ru

I spent three iterations on an auto-fix pipeline that still doesn't work reliably. Here's what I learned. Loop 1 Wrote a background script. Pull tickets from Azure DevOps, run them through a local model, hand to a coding agent, push the result. Poll → triage → fix → push. Worked 40% of the time on trivial tickets. Anything that crossed file boundaries or needed real context — stalled or hallucinated. I shipped it anyway. That was naive. Loop 2 Made it smarter. Pre-selected relevant files. Broke big tickets into subtasks. Turned complex edits into atomic steps with verification between each. Got it to 55% or so. But every fix created two new edge cases. The complexity was compounding faster than the reliability. Loop 3 Went all in. Embeddings for dedup. Multi-repo routing. Auto-revert. A learning loop that fed failures back into future runs. The model server started dying. 890 memory errors in a day. Root cause: two independent consumers hitting the same local model server, each with its own retry loop. When memory filled up, retries amplified instead of staggering. The system was making itself worse. Fixes were simple in hindsight — stop retrying OOM, serialize access, use the local binary not npx. But the pattern kept repeating: add more to fix the last thing, break something else. Where I'm At The pipeline still only works on easy tickets. Hard ones need a human. After three rounds, the main thing I learned is that local models hit a wall before your ambition does — not in quality, in working memory. And adding features doesn't fix reliability gaps. It just moves them around. The 507 retry spiral taught me more than any successful deploy this year. Because it was entirely my fault. Not the model's, not the framework's. I built concurrent consumers with independent retry loops and expected them to coordinate. They didn't. What's Next I'll do a fourth loop. Smaller. A dedicated fast model for cheap work, the big model only for editing. One consumer at a time. Might

TL;DR — When the spec is good, implementation needs less model. I started using a top-tier model to write the spec and a cheaper, faster one to implement it — still using the strong model, just spending it on the spec instead of the implementation. The gain isn't some magic prompt phrasing; it's the context: explicit business rules, audited project constraints, a defined output contract. That's systems engineering — the discipline of anyone who's kept real software alive, whatever their stack. Every backend dev knows the scene: the Swagger is out of date, the last hotfix shipped without a unit test, and the README.md documents a command nobody's used in six months. The code works. The docs lie. And the gap between the two is exactly where AI — and we — start to go wrong. I've spent the last few months developing with AI for real inside production projects, not tutorial greenfield. My takeaway was less about which model to use and more about a shift that already has a name: the move from prompt engineering to context engineering . The difference isn't semantic. Prompt engineering treats the problem as writing — finding the magic phrase. Context engineering treats it as what it always was: a systems engineering problem . And it's where my backend background applied most directly — though anyone who's kept a real system alive has the same instinct. The experiment that convinced me Let me start with the evidence, because that's what made me take this seriously. My reflex, for a long time, was to reach for the strongest model for everything — more expensive, smarter, fewer errors. Makes sense on paper. In practice, I saw something else. When the task's specification is well done — explicit business rules, audited project constraints, a defined output format — the model capability needed for implementation drops sharply. Enough to split the work by stage: I started using a top-tier model (currently Opus) to write the spec , and a cheaper, faster model (Sonnet) to implemen

Disclosur: Ini dari tim Nexotao — saya bahas gateway kami sendiri di bawah. Saya jaga sebatas fakta yang bisa kamu cek sendiri: semua nama model, context window, dan harga ada di halaman pricing kami, dan saya kasih linknya. Kalau kamu developer di Indonesia, kemungkinan besar pernah kejedot ini: API OpenAI dan Anthropic minta kartu kredit luar negeri . Nggak punya kartu, nggak bisa pakai API. Banyak dari kita mentok di situ. Solusi yang jalan sekarang: gateway lokal yang nerima QRIS / Rupiah . Ini versi jujurnya — gimana cara kerjanya, berapa biayanya, dan apa yang belum bisa. Dua model live, satu API yang kompatibel Lewat Nexotao kamu pakai dua model teks: Claude Opus 4.8 ( claude-opus-4-8 ) — context window 350.000 token DeepSeek-V4-Pro — context window 131.072 token Itu angka context window yang dipublikasikan apa adanya — tanpa pemotongan diam-diam. Endpoint-nya kompatibel dengan OpenAI dan Anthropic , jadi biasanya cukup ganti base URL sama key-nya. Format OpenAI: from openai import OpenAI client = OpenAI ( base_url = " https://api.nexotao.com/v1 " , api_key = " sk-nexo-... " ) resp = client . chat . completions . create ( model = " claude-opus-4-8 " , messages = [{ " role " : " user " , " content " : " Halo " }], ) print ( resp . choices [ 0 ]. message . content ) Format Anthropic: curl https://api.nexotao.com/v1/messages \ -H "x-api-key: sk-nexo-..." \ -H "anthropic-version: 2023-06-01" \ -H "Content-Type: application/json" \ -d '{"model":"claude-opus-4-8","max_tokens":256, "messages":[{"role":"user","content":"Halo"}]}' Cara bayarnya Top up saldo Rupiah via QRIS , mulai Rp10.000 . Tanpa kartu luar negeri. Bayar sesuai pakai — dipotong per token. Tanpa langganan , dan saldo nggak hangus. Tiap response ada header X-Cost-Rp , jadi kamu lihat biaya rupiah persis tiap request. Berapa biayanya Saat tulisan ini dibuat, Claude Opus 4.8 lewat gateway sekitar 70% lebih murah dari harga retail resmi (input) — tapi jangan percaya saya gitu aja. Halaman perbandingan har

Disclosure: This is the Nexotao team — I'm describing our own gateway below. I've kept it to facts you can verify yourself: every model name, context window, and price here is on our live pricing page, and I link it. If you're an Indonesian developer, you've probably hit this wall: the OpenAI and Anthropic APIs want a foreign credit card . No card, no API. A lot of us get stuck right there. The fix that works today: a local gateway that takes QRIS / Rupiah . Here's the honest version of how it works, what it costs, and what it doesn't do. Two live models, one compatible API Through Nexotao you call two text models: Claude Opus 4.8 ( claude-opus-4-8 ) — context window 350,000 tokens DeepSeek-V4-Pro — context window 131,072 tokens Those are the real, published context windows — no silent truncation. The endpoint is OpenAI- and Anthropic-compatible , so you usually just change the base URL and key. OpenAI format: from openai import OpenAI client = OpenAI ( base_url = " https://api.nexotao.com/v1 " , api_key = " sk-nexo-... " ) resp = client . chat . completions . create ( model = " claude-opus-4-8 " , messages = [{ " role " : " user " , " content " : " Hello " }], ) print ( resp . choices [ 0 ]. message . content ) Anthropic format: curl https://api.nexotao.com/v1/messages \ -H "x-api-key: sk-nexo-..." \ -H "anthropic-version: 2023-06-01" \ -H "Content-Type: application/json" \ -d '{"model":"claude-opus-4-8","max_tokens":256, "messages":[{"role":"user","content":"Hello"}]}' How you pay Top up your Rupiah balance via QRIS , from Rp10,000 . No foreign card. Pay-as-you-go — deducted per token. No subscription , and the balance never expires. Every response carries an X-Cost-Rp header, so you see the exact rupiah cost of each request. What it costs At the time of writing, Claude Opus 4.8 runs roughly 70% below official retail input pricing through the gateway — but don't take my word for it. The comparison page shows live per-token rates and computes "vs official" automati

Earlier this month, a German court ruled that Google is liable for its AI search summaries. Rejecting defenses like “users can check for themselves,” and that they generally know “that information generated with AI should not be blindly trusted,” the court held that the AI’s summaries are reflections of the company and “above all an expression of Google’s business activities.” This is the latest skirmish in a decades-old battle over internet publishing. Historically, there were two different types of information distributors: carriers and publishers. A phone company is a carrier. It’ll transmit whatever you say, even discussions about committing a crime. Words are words, and the phone company does not know—nor is it liable for—the words you choose to speak. A newspaper, on the other hand, is a publisher. It decides the words it publishes, and what quotes to include in its articles. If those words or quotes are defamatory or otherwise illegal, it’s liable...

AI harnesses are the new vendor lock-in. To swap across harnesses easily without rewriting your app, LiteLLM launched the Lite-Harness SDK . Run your prompt across different harnesses: from lite_harness import query , AgentOptions prompt = " Fix the failing test " # Claude Code harness async for message in query ( prompt = prompt , options = AgentOptions ( harness = " claude-code " , model = " claude-opus-4-8 " ), ): print ( message ) # Codex harness async for message in query ( prompt = prompt , options = AgentOptions ( harness = " codex " , model = " gpt-5.5 " ), ): print ( message ) To enable cost controls, fallbacks, and logging, point it to your LiteLLM AI Gateway: export LITELLM_API_BASE = https://litellm.your-company.com/v1 export LITELLM_API_KEY = sk-litellm-... Engineer's Takeaway: This SDK unifies how you invoke the agents, not how they run internally. Each harness keeps its native loop and tool-calling semantics. It is perfect for A/B testing agent performance and centralizing costs, but remember it is in public beta, so custom tool injection might require extra work! The Problem I Had My team was building an internal bot to fix failing CI/CD tests. We had three engineers advocating for three different harnesses: one wanted Claude Code, another Codex, and another Pi AI. Without an abstraction layer, we would have had to maintain three forks of the same bot , with three different SDKs, three logging systems, and three ways to track costs. It would have been an impossible maintenance burden. How Lite-Harness Helped The SDK solved that exact pain point in three concrete dimensions : 1. Unified Invocation (Time Savings) Instead of maintaining three separate implementations, I had a single query() that routed to whichever harness I wanted. Switching from Claude Code to Codex was literally just changing a string in the options. This allowed us to do real A/B testing in production for two weeks without rewriting any core logic. 2. Cost Observability (The Killer

TL;DR: An API gateway manages HTTP traffic between services — auth, routing, rate limiting, load balancing for REST and gRPC. An AI gateway manages LLM workloads — token-based rate limiting, model routing, cost attribution, semantic caching, guardrails. Use an API gateway for your microservices. Use an AI gateway for your LLM traffic. Most production teams eventually need both, sitting at different layers. This post walks through exactly where each one fits. When we started adding LLM features to our platform, we already had Kong running for our microservices. The instinct was natural: route the LLM traffic through Kong too. Same auth, same rate limiting, same observability stack. One gateway to rule them all. It worked — for about six months, and only in the sense that requests got through. What it didn't give us was anything useful for actually managing AI workloads. We had no idea what each team was spending on tokens. We had no way to set a budget cap that would fire before the bill arrived. Our rate limits were based on requests per minute, which meant a single request with a 50k token prompt counted the same as one with a 200 token prompt. And when OpenAI had a partial outage, Kong had no concept of "try Anthropic instead" — we just served errors. None of that is a criticism of Kong. It's doing exactly what it was designed to do. The problem was us expecting an API gateway to handle a fundamentally different category of infrastructure problem. Here's the precise distinction, and why it matters architecturally. What an API gateway actually does An API gateway is a reverse proxy that sits between client applications and backend services. It handles the cross-cutting concerns of service-to-service HTTP communication: authentication, authorization, rate limiting, load balancing, SSL termination, request transformation, and routing based on URL paths or headers. A typical request flow through an API gateway: Client sends a request to the gateway endpoint Gateway ve

At least one malware developer is adding text about nuclear and biological weapons to their spyware, in an effort to stop automatic AI analysis. Details : The _index.js payload begins with a large JavaScript block comment containing fake system instructions and policy-triggering content. Because it is inside a comment, it does not affect JavaScript execution. The runtime skips it. The real malware begins after the comment with a try{eval(…)} wrapper around a large character-code array and a ROT-style substitution function. This header appears designed for AI-mediated analysis, not for Node, Bun, or Python. It attempts to derail scanners or analyst copilots that feed the beginning of a file to a language model without clearly isolating the content as untrusted data. In weak pipelines, this can cause refusal behavior, prompt confusion, context pollution, or premature classification before the scanner reaches the actual malware...

TL;DR An AI gateway is a middleware layer between your application code and your LLM providers - it centralises routing, auth, rate limiting, cost tracking, and guardrails in one place You probably don't think you need one until something specific breaks: a runaway cost spike, a failed model causing silent errors, a security audit you can't pass We went from scattered SDKs and shared API keys to a gateway-first setup over about three months - this post covers what changed and what we'd do differently Six months ago we had what I'd describe as a functional mess. We were running three LLM providers - OpenAI for our customer-facing chat, Anthropic for internal document summarisation, and a self-hosted Llama model for batch classification jobs. Each had its own SDK. Each had its own API key, living in .env files on whoever's machine had last run that service. Each had its own rate limiting logic, copy-pasted between services with slight variations. It worked, in the way that things work when nobody has had a bad enough incident yet. The incident arrived on a Tuesday. A background job that was supposed to run once a week got accidentally scheduled to run every minute. It was calling GPT-4o. We noticed when the Slack alert fired at 2am about an unusual credit card charge. By the time someone killed the job, we'd burned through $340 in about four hours. The API key had no spending limit. There was no alerting on token usage. The job had no rate limiting. All three of those gaps were things we knew about and hadn't prioritised. That week, we started properly looking at AI gateways. What an AI gateway actually is? The simplest definition: an AI gateway is a middleware layer that sits between your application code and your LLM providers. All your LLM requests go through it, and it handles the cross-cutting concerns that you'd otherwise have to re-implement in every service: routing, authentication, rate limiting, cost tracking, caching, fallbacks, guardrails. The analogy that

How to Access DeepSeek API from Outside China (2026 Guide) DeepSeek has quietly become one of the best open-weight LLM families available. Their V4-Pro model matches GPT-4o within 3-5% on coding benchmarks (HumanEval, MBPP) while costing roughly 90% less per token. The problem? Actually getting access as an overseas developer. The Registration Wall If you try to sign up for DeepSeek's official API directly, you'll hit this: ✕ +86 phone number required for SMS verification ✕ Alipay or WeChat Pay only — no Stripe, no PayPal ✕ Documentation is primarily in Chinese ✕ VPN required and it drops mid-request ✕ Different auth system than OpenAI This isn't a minor inconvenience — it's a hard blocker for most overseas developers. I spent a full weekend trying to work around it before finding a solution that actually worked for production use. Option 1: DIY Proxy (Not Recommended) You could technically set up a Chinese VPS as a relay, register through a Chinese friend's number, and proxy requests. I tried this approach. Problems: Your Chinese VPS adds 100-300ms latency You're responsible for keeping the integration working If your Chinese friend's number gets flagged, you're locked out No SLA, no support, no monitoring Payment still requires Alipay — you need a Chinese bank account or a friend After a weekend of futzing with this, I abandoned it. Not production-ready. Option 2: Third-Party Gateway (What I Use) There are now services that handle the China-side complexity and expose DeepSeek through a standard OpenAI-compatible API. They handle: Chinese phone number verification Alipay/WeChat payment (you pay via Stripe instead) API routing with global edge caching Load balancing across multiple Chinese providers Setup is literally two lines: # Before: Direct OpenAI client = OpenAI ( base_url = " https://api.openai.com/v1 " , api_key = OPENAI_KEY ) # After: Via gateway client = OpenAI ( base_url = " https://api.tokenmaster.com/v1 " , api_key = TM_KEY ) That's it. Same SDK, same i