You've deployed your app to Kubernetes. The pod starts — then it gets killed. Or it's running but no traffic reaches it. Or it takes 90 seconds to initialize and gets restarted in a loop. Every one of these problems traces back to the same root cause: misconfigured or missing health probes . Kubernetes gives you three types of probes: livenessProbe , readinessProbe , and startupProbe . Each serves a different purpose. Mix them up and your pods restart in infinite loops. Get them right and your deployments self-heal, scale correctly, and handle rolling updates without a single dropped request. Here's what each probe does, when to use it, and how to configure it for a real production service. 1. Liveness Probe: Is the Container Alive? The liveness probe answers one question: "Is this container still running correctly?" If the probe fails, kubelet kills the container and restarts it. livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 failureThreshold: 3 Use liveness probes for deadlock detection . If your app enters a state where it's alive but not making progress (a goroutine leak, a stuck mutex, an infinite loop), the liveness probe exposes that and triggers a restart. The #1 mistake people make: using the liveness probe to check external dependencies like databases or upstream APIs. Don't do this. If your database is down and your liveness probe fails, Kubernetes will restart your pod — but the database is still down. Restarting the app doesn't help, and now you have a crash loop on top of a DB outage. That's worse. Liveness probes should only check internal process health. Not database connectivity, not Redis, not upstream services. 2. Readiness Probe: Is the Container Ready for Traffic? The readiness probe answers: "Should this pod receive traffic?" If it fails, the pod is removed from all Service endpoints. It is not restarted. readinessProbe: httpGet: path: /ready port: 8080 periodSeconds: 5 failureThreshold: 2 successThre

A previous post covered how to absorb PHP 8.2 Deprecated warnings from WP-CLI using a three-layer defense . The approach — prepending WP_CLI_PHP_ARGS to set error_reporting — works in many environments. But a case came up where Deprecated warnings wouldn’t disappear despite the same configuration. Tracing the cause revealed a structural reason why the environment variable never arrived. This post records that root cause and the three-part fix added in v1.6.8. Why environment variables don’t arrive — the phar + shebang execution path An agency reported that on Xserver, plugin list retrieval was failing across multiple sites (referred to here as "site A / site B") with a large volume of Deprecated messages. We reproduced the same behavior on our own Xserver setup (PHP 8.2.30, WP-CLI 2.7.1) and traced the execution path. Xserver’s /usr/bin/wp is a phar binary. Inside, it starts with a #!/usr/bin/env php shebang, so the actual startup sequence looks like this: shell → /usr/bin/wp (shebang: #!/usr/bin/env php) ↓ env locates php and starts it ↓ php loads the phar → WP-CLI runs In this path, WP_CLI_PHP_ARGS is never read as a PHP startup option. WP_CLI_PHP_ARGS is supposed to let WP-CLI pass a -d flag to PHP, but when PHP itself is launched via shebang, control never reaches the point where WP-CLI can inject that flag into PHP’s invocation. # doesn’t work — /usr/bin/wp on Xserver is a shebang-launched phar WP_CLI_PHP_ARGS = "-d error_reporting='E_ALL & ~E_DEPRECATED'" wp plugin list --format = json # works — -d goes directly to the php binary php -d error_reporting = 'E_ALL & ~E_DEPRECATED & ~E_USER_DEPRECATED' /tmp/wp-cli-2.7.1.phar plugin list --format = json We verified this against our production setup: with the first form, 407 Deprecated lines remained; with the second, 0. Pillar A — detecting the php-direct path and injecting -d The fix: inspect wp_cli_path for whether it’s a php-direct invocation, and if so, inject -d error_reporting immediately after the PHP

Generate Topologic contours Discussion | Link

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry . This is our official submission for the DEV Big Summer Bug Smash challenge under the #bugsmash track. Below is the technical tale of how we isolated, debugged, and optimized cross-layer node latency issues when deploying our Web3 framework on Polygon. The Problem: The Post-Hard Fork RPC Latency Wall 🐛 During heavy network volume spikes or directly following major ledger upgrades, our automated event listener logging pipeline kept crashing with random, non-deterministic invalid block range exceptions when attempting to pull historical data blocks via standard eth_getLogs routines. The Technical Root Cause The root bottleneck came down to an internal desync inside shared public RPC telemetry environments: The Bor Layer mints new block headers at a blistering speed (~2 seconds). The Internal Indexer DB takes slightly longer to completely unpack, parse, and commit transaction event logs to disk. When our asynchronous scripts called the node, latest grabbed the bleeding edge tip of the chain from memory, but a simultaneous getLogs query hit the slower indexer database. This split-millisecond race condition threw immediate pipeline errors. The Fix: Layered Application Buffering 🛠️ To smash this bug without modifying low-level node client builds, we engineered a programmatic block-padding delay loop directly into our interaction routers. Instead of tracking unfinalized tip block states blindly, we forced our queries to target safe block ranges sitting securely just behind the tip of the chain. // Localized block-buffer deployment fix const currentChainTip = await provider . getBlockNumber (); const indexedBlockBoundary = currentChainTip - 3 ; // Buffer 3 blocks (~6 second safety zone) const targetLogs = await contract . getLogs ({ fromBlock : indexedBlockBoundary - 20 , toBlock : indexedBlockBoundary }); This structural adjustment completely stabilized our off-chain reward data pipeline, gua

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

I've been working in software for more than 15 years. This is the first piece I've decided to write about the job. I picked this topic because it was a small, real problem that cost me time more than once, until I finally stopped and fixed it properly. Quick disclaimer: this isn't distributed tracing or multi-service observability. It's a targeted fix for one specific problem: correlating logs within a single Step Function execution. If your scenario is different, the problem takes a different shape. What matters here is the reasoning, not the recipe. Quick context for anyone who doesn't work with serverless: AWS Step Functions is an orchestration service. You design a workflow as a state machine (JSON), and each state usually triggers an AWS Lambda (a function that runs on demand, no server to manage). In my case, a request kicks off a state machine execution that passes through several Lambdas in sequence: some do validation, others call external services, and one waits on a callback response before moving on. The fix turned out to be simpler than it sounds: a field Step Functions already offered for free, and I just wasn't using it. A request was processed 120 days ago. The customer calls: "Did you receive my request? I never heard back." You open the AWS Console. The Step Functions execution is already gone. Execution history only sticks around for 90 days, and even within that window, finding one specific execution among thousands is already a hassle. That leaves CloudWatch, with logs from every Lambda in the pipeline. You have one piece of data: the case ID. But every Lambda logged its own way: no standard, no correlation between them. To reconstruct what happened, you have to open each function's logs in the order the workflow probably ran, and piece it together by hand. It wasn't the error itself that hurt. It was the time lost just organizing the logs before you could actually start investigating. The Chaos The pipeline wasn't huge, but it wasn't trivial ei

The openFDA enforcement API is free, keyless, and well documented on the surface. It is also full of failure modes that return HTTP 200 with quietly wrong data. Every number and error string below was measured against the live API on 2026-07-20; anything I could not reproduce has been cut. Pick the right endpoint first There are four recall-shaped endpoints and they are not interchangeable. Choosing wrong gives you a different universe of records with no warning. Endpoint Records What it is drug/enforcement.json 17,793 Recall Enterprise System (RES) drug recalls device/enforcement.json 39,519 RES device recalls food/enforcement.json 29,224 RES food recalls device/recall.json 58,756 CDRH device recall database, a different schema entirely The three enforcement endpoints share their schema. device/recall.json does not: its fields include cfres_id , product_res_number , k_numbers , root_cause_description , event_date_posted , event_date_terminated and recall_status , and it has no classification field at all ( count=classification.exact returns HTTP 404 "Nothing to count" ). If you are filtering for Class I, you want an enforcement endpoint. The two families also refresh on different clocks. On 2026-07-20 the three enforcement endpoints reported meta.last_updated of 2026-07-08, while device/recall.json reported 2026-07-17. The OR bug that silently returns the wrong answer This is the single most expensive trap, and it is undocumented. An unparenthesized OR discards every clause except the last one. All figures below are from food/enforcement.json . search=classification:"Class I" OR state:"CA" returns 4,003 - exactly the count for state:"CA" alone. Reverse the operands and you get 12,809 - exactly classification:"Class I" alone. Wrap it: search=(classification:"Class+I"+OR+state:"CA") returns 14,822 in both orders. That is the real union (12,809 + 4,003 - 1,990 overlap, and the AND of the two clauses does return 1,990). No error is raised in any case. The nastier varia

The final approval settles one case, but it doesn't resolve the broader issue of using copyrighted works to train AI models.

David Vélez and Robin Vince join the boards of the OpenAI Foundation and OpenAI Group PBC, bringing global leadership in finance, technology, and governance.

The de-extinction startup is looking to double or triple its previous valuation, according to the report.