今日已更新 311 条资讯 | 累计 29453 条内容
关于我们

标签:#p

找到 12189 篇相关文章

AI 资讯

Medir si un LLM nombra a tu empresa: por qué una captura no sirve como métrica

Cada vez más gente arranca la búsqueda de un proveedor preguntándole a un modelo en vez de a un buscador. Y no pide diez opciones para comparar: pide una recomendación y recibe dos o tres nombres. Si tu empresa no está ahí, no quedaste octava. No estás en la respuesta. La pregunta que sigue es obvia: cuánto tarda en cambiar eso. Pero antes hay un problema más aburrido y más importante, que es cómo se mide. Lo escribo porque es la parte que casi nunca se cuenta y es donde se rompen los informes. Una captura de pantalla no es una medición Es el error más común y el más difícil de discutir, porque la captura parece prueba. La respuesta de una app conversacional depende del historial de la cuenta, de la sesión, del ruteo interno del proveedor, de si esa consulta activó búsqueda web o no, y de la región desde donde se pregunta. Dos personas preguntando lo mismo el mismo día reciben respuestas distintas. La misma persona preguntando dos veces también. O sea: la salida no es determinista y el instrumento no es estable. Una captura te dice qué pasó una vez, en un contexto que no podés reconstruir. Como métrica de seguimiento no sirve para nada. Lo que sí sirve es una serie: la misma consulta, literal, contra el mismo motor, con el mismo criterio de clasificación, repetida en el tiempo. El valor absoluto de un punto importa poco. Lo que importa es la diferencia entre puntos. Fijar el texto de la consulta, no la etiqueta Este es un bug de proceso que da resultados verosímiles y falsos. Si guardás en la planilla una etiqueta como "consulta de chatbot" en vez del texto exacto que preguntaste, dentro de dos meses nadie se acuerda del wording. Y el wording cambia el resultado: preguntar "quién hace X en Argentina" y "mejores empresas de X en Argentina" devuelven listas distintas. Cuando el texto se corre entre rondas, la serie deja de ser comparable, pero el gráfico sigue dibujándose igual de lindo. Guardá el string literal, versionado. Si tenés que cambiar una consulta, empezá u

2026-08-05 原文 →
AI 资讯

How Much Does It Cost to Self-Host Open Models on AWS?

Your AI bill tripled last quarter. Your CTO forwarded you an article about companies saving 70% by switching to open models. Now someone is asking you to figure out what that would actually look like. I spent the last few weeks digging into this. The numbers, the hardware, the real trade-offs. Here's what I found, with enough specifics that you can actually make a decision rather than just nodding along to another "open source is the future" think piece. What "Open Models" Actually Means When someone says "open model" they mean an AI model where the weights (the learned parameters that make the model work) are publicly downloadable. You grab the file, run it on your hardware, and you don't pay anyone per request. The big names right now: Meta's Llama 4, DeepSeek V4, Zhipu's GLM-5.2, Moonshot's Kimi K3, Alibaba's Qwen 3.5, and Google's Gemma 4. These aren't toys. Some of them genuinely compete with the frontier models on real benchmarks. Chinese open models now handle over 30% of enterprise traffic on OpenRouter, up from 4.5% in early 2025. That's a massive shift in barely a year. The Architecture: What You Actually Need You want your team to use an open model. Here's the stack, from bottom to top. Hardware (The Expensive Part) A model is a giant file. We're talking anywhere from 4 GB (a small 7B model, quantized) to 1.5 TB (Kimi K3, full weights). That entire file needs to sit in GPU memory to run fast. Why GPU memory specifically? Because generating each word in a response requires billions of multiply-and-add operations. GPUs do thousands of these in parallel. A CPU does them one at a time. The practical difference: a 7B model on a CPU generates 2-5 tokens per second (painfully slow for interactive use). The same model on a GPU generates 30-80 tokens per second (feels instant). For one person on a CPU, it might be tolerable. For a team of 10 all hitting the same endpoint? Unusable. Requests queue up and everyone waits 30-60 seconds for responses. Think of it like

2026-08-05 原文 →
AI 资讯

One Rails request, one event: production context for coding agents

Wide Events is a Rails gem that puts the production context a coding agent needs onto one OpenTelemetry root span per request or job. In one production search request, the root event showed 30.0 seconds total duration, 446 ms of Postgres time, and 29.4 seconds of outbound HTTP time. That was enough to focus the investigation on an external dependency. The trace then identified a POST that took 28.9 seconds. The trace contained 82 spans and 20,261 bytes of attribute JSON. The root event contained 40 attributes and 1,420 bytes. This is not a token benchmark, but it shows why the root event is a more compact starting point for an agent. Agents can read the code, but not the running system A coding agent starts with an unusual advantage: it can search every model, controller, job, migration, and test in a few seconds. It also starts with a serious blind spot. The repository cannot tell it: which account experienced the problem which build was running which feature-flag variant was active how many queries the request issued whether a semantic-search leg degraded how much an LLM call cost whether the same symptom appears in one tenant or every tenant Those answers often exist somewhere, but “somewhere” might mean a trace waterfall, application logs, a feature-flag service, product analytics, and a database console. Pulling all of that into a context window is expensive and usually requires several joins that were never designed in advance. A wide event changes the starting point. The app accumulates the context it learns while processing one unit of work, then attaches the completed flat map to the OpenTelemetry root span. The span is marked main=true , so every request or job can be queried as one row. request or job -> Rails and domain context accumulate -> child spans contribute dependency counts and timings -> one flat map is flushed onto the root span -> ClickHouse stores one queryable row The trace still exists. Wide Events gives it an application-shaped index. If y

2026-08-05 原文 →
AI 资讯

I Built a Server Agent Because Uptime Checks Tell You What Failed, Not Why

A status page has a blind spot. It can tell you that your API is returning 502s. It can tell you that a TCP port stopped accepting connections. It can tell you when the incident started. It usually cannot tell you why . Was the application host out of memory? Was disk I/O saturated? Did load climb for 40 minutes before users noticed? Was the server completely healthy and the real problem somewhere else? Those answers often live in a separate monitoring product, disconnected from the incident timeline and disconnected from the status page. That is why I built Servers for StatusPage.me. It is a small, customer-installed host metrics agent and dashboard. You install it on a machine you operate, and it reports CPU, memory, swap, load, disk, and network metrics back to your account. The important part is not “now there are more graphs.” The important part is seeing an outage and the host evidence around it on the same timeline. External checks answer one question. Host metrics answer another. Regular uptime monitoring is still the right tool for the outside-in view: Can users reach the website? Is the API returning the expected response? Does DNS resolve correctly? Is the database port open? Did a scheduled job run? But those checks do not run inside your infrastructure. A healthy HTTP response does not prove that a background worker is about to run out of memory. A timeout does not prove that the app server is overloaded. And an incident can start with a slow disk or growing swap usage long before an endpoint is fully unavailable. The distinction is simple: External monitoring tells you what users can see. Host metrics help explain what the machine was doing when they saw it. You need both. What Servers includes Each registered host gets a dedicated dashboard page with: CPU user, system, and I/O wait utilization Memory use Swap use Load averages Disk use and read/write throughput Network inbound and outbound throughput A human-readable OS description for account owners

2026-08-05 原文 →
开发者

I got tired of mocking Date, so I built a TimeProvider for TypeScript

Every (or at least a lot of) project seems to have code like this somewhere: if (user.subscriptionEndsAt < new Date()) { // ... } There's nothing wrong with it... until you have to test it. Then you end up freezing time, mocking Date, enabling fake timers, remembering to restore them afterwards, and hoping another test didn't leave the clock in a weird state. While Jest's and Vitest's fake timers are great tools, they always felt like they were solving the problem from the outside by patching global APIs. I wanted to try something different. Time is a dependency When you think about it, the current time isn't much different from a database or an HTTP client. Your business logic depends on it, but it doesn't have to know where it comes from. Instead of writing this: const now = new Date(); what if we wrote this? const now = timeProvider.now(); Suddenly, testing becomes boring—in the best possible way. You don't need global fake timers anymore. You just pass a different implementation. .NET had the same idea While looking into this, I discovered that .NET 8 introduced a TimeProvider abstraction. Seeing that was reassuring. It suggested I wasn't the only one who felt that "current time" deserved to be treated as a real dependency. I didn't want to copy the .NET API, but I did like the underlying idea. So I started building a version that felt natural in the TypeScript ecosystem. It grew beyond a clock At first I only wanted to replace new Date(). Then I realized the same issue exists with setTimeout, setInterval, performance measurements, and a few other APIs. They all depend on the environment's notion of time. So the library slowly became an abstraction around all of those instead of just "what time is it?". Is this actually useful? That's the part I'm still curious about. In the projects I've worked on, I prefer injecting time over patching globals during tests. Maybe other teams have reached the same conclusion. Maybe everyone is perfectly happy with fake timers an

2026-08-05 原文 →
AI 资讯

How to Actually A/B Test AI Avatar vs. Text Chat Conversion (A Technical Approach)

Following up on a common claim in the AI avatar space — that voice/video avatars convert better than plain text chat — there's surprisingly little rigorous testing behind it. If you're building or embedding one of these widgets, here's a practical way to actually measure it instead of trusting vendor case studies. Why This Is Harder Than a Normal A/B Test Standard A/B testing swaps one variable (a button color, a headline) while holding everything else constant. Avatar vs. text chat isn't that clean — you're changing interaction modality, response latency expectations, and visual real estate simultaneously. You need to isolate the variable that actually matters: does voice/video presence drive conversion, independent of the underlying conversation quality? A Cleaner Experimental Setup javascript // Pseudocode for variant assignment function assignVariant(sessionId) { const hash = hashSessionId(sessionId); return hash % 2 === 0 ? 'avatar' : 'text'; } Key controls to hold constant across both variants: Same LLM backend and prompt/knowledge base — the conversation logic shouldn't differ, only the presentation layer Same lead capture form and CTA placement — don't let UI differences beyond avatar-vs-text confound the result Same traffic source — segment by acquisition channel if traffic mix varies, since paid vs. organic visitors convert differently regardless of chat UI Minimum sample size before evaluating — novelty effects are real; running this for 3 days will overstate the avatar's lift. Run for at least 2-3 weeks to let novelty decay. Metrics to Track (Not Just Conversion Rate) Conversion rate alone hides why one variant wins or loses: session_start_to_first_message (engagement friction) message_count_per_session (depth of interaction) time_to_form_completion (avatar/video adds latency — does it cost or gain time?) bounce_rate_before_first_response lead_quality_score (if you can grade downstream — a lead isn't a conversion if it's junk) A common finding worth watc

2026-08-05 原文 →
AI 资讯

How EvalPort's Grader System Works: 11 Types for LLM Evaluation

How EvalPort's Grader System Works When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own way of scoring LLM outputs — DeepEval uses metric classes, Promptfoo uses assertion objects, Inspect AI uses solver functions. We needed a system expressive enough to cover 90%+ of real-world eval needs, but simple enough that any framework could implement it. The result: 11 grader types that carry their own semantics. A grader isn't just a name — it specifies its parameters, its model, its threshold. An eval suite is self-describing. The 11 Grader Types exact_match — Compare output to expected output, optionally ignoring case. contains — Check if the output contains a substring. regex — Match against a regular expression. semantic_similarity — Embed output and expected output, compare cosine similarity against a threshold. llm_judge — Use an LLM to evaluate the output against a prompt template. The most powerful grader. json_schema — Validate that the output is valid JSON matching a JSON Schema. json_path — Extract a value from JSON output using a JSONPath expression, then compare it. code — Run a function to evaluate the output. human — Defer to human review. model_graded — Compare the output to a reference answer using a model. custom — Escape hatch for graders not covered by built-in types. How Graders Connect to Test Cases A test case references graders by ID. Multiple graders can evaluate the same test case. The ResultSet records each grader's score separately. Why This Design Works Self-describing: An eval suite carries everything a framework needs to execute it. Framework-agnostic: Any framework can implement any subset of grader types. Extensible: The custom type lets frameworks bring their own graders. Comparable: Results from different frameworks use the same grader IDs. Try It pip install evalport-sdk npm install evalport-sdk Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md Repo: https://gith

2026-08-05 原文 →