今日已更新 240 条资讯 | 累计 23823 条内容
关于我们

There was no independent, measured view of AI-API latency by region — so I built one

Max Bob 2026年07月24日 02:19 0 次阅读 来源:Dev.to

If you build anything on top of hosted AI APIs, latency isn't a detail you get to ignore — it's a feature. A sluggish time-to-first-token is the difference between an assistant that feels alive and one that feels broken. Yet when I went looking for an honest answer to a simple question — how fast is provider X from where my users actually are? — I couldn't find one. The numbers people quote tend to come from a single machine in a single region (usually somewhere in the US), from vendor-reported status pages, or from a benchmark that got run once and never refreshed. There was no independent, measured , regional view. So I built one: LLM Latency Tracker , a provider-neutral tracker of latency and uptime for AI inference APIs. How it works The core idea is boring on purpose: actually measure, don't scrape. A small Python prober — standard library only, no API key needed for the edge probes — opens real connections to each provider's endpoint and times every phase of the handshake: DNS resolution → TCP connect → TLS negotiation → time-to-first-byte (TTFB) . That's the edge view: how long the network path itself takes before a single byte comes back. Separately, where possible, it measures inference time-to-first-token (TTFT) — the thing your users actually feel, i.e. how long after you hit "send" the model starts streaming. Those two numbers answer different questions, and keeping them apart matters. Edge latency is about the network and the front door; TTFT is about the model and the queue behind it. The probes run from four regions — Europe (Germany), US Central, Asia (Tokyo), and South America (São Paulo) — because "fast" is meaningless without "from where." Results land in a SQLite time-series; a static-site generator turns that into the pages you see, hosted on Cloudflare Pages, and the whole thing is self-updating on a schedule. There's no always-on backend to rot or page me at 3am. It currently covers ~45 providers — the usual Western labs (OpenAI, Anthropic, Go

本文内容来源于互联网,版权归原作者所有
查看原文