标签：#observability

I have a ritual. Whenever a pipeline latency alert hits my phone, my first instinct isn't to open a heavy dashboard or spin up a full Grafana instance. I grab my terminal and start firing curl commands at port 9600. curl -s localhost:9600/_node/stats?pretty ... curl -s localhost:9600/_cat/pipelines ... curl -s localhost:9600/_plugins . It's a repetitive, mindless sequence of commands. It works, but it's reactive and solo. You are the one parsing the JSON, you are the one looking for the pattern in the JVM heap usage, and you are the one manually correlating a spike in event flow with a specific thread lock. With the Model Context Protocol (MCP), that ritual is becoming obsolete. I've been experimenting with connecting MCP-compatible agents—specifically through Cursor and Claude—directly to Logstash via a specialized API server. The difference isn't just 'convenience.' It's an architectural shift from manual inspection to agentic triage. Moving beyond the Chatbot Most people treat AI like a documentation search engine. They ask, "How do I configure a JDBC input in Logstash?" That’s fine, but it doesn't help when your production cluster is turning 'yellow' at 3 AM. The real value of MCP isn't the ability to talk to an AI; it's the ability to give that AI a set of hands—specifically, a set of tools that can interact with live infrastructure. I recently integrated the Logstash Server-side Log Pipeline API into my workflow. This isn't some experimental script I wrote over a weekend; it’s a production-grade implementation built on MCPFusion. It gives an AI agent direct access to several critical Logstash endpoints through a controlled, sandboxed environment. The Triage Workflow: A Real Scenario Let's walk through how this actually changes the debugging loop. Imagine you have a spike in ingestion lag. In the old way, you’d be digging through terminal history. In the new way, your agent acts as an extension of your SRE toolkit. 1. Initial Health Check Instead of parsing raw

GitHub announced enterprise-managed OpenTelemetry export for Copilot activity from VS Code and Copilot CLI on July 8, 2026. Primary source: GitHub Changelog, July 8, 2026 . Export availability is only the start. The receiving collector becomes an enterprise ingress point. This is an unexecuted operating plan; signal types, attributes, endpoint requirements, and controls must be checked against current GitHub documentation. Isolate the path managed clients -> private telemetry ingress -> dedicated OTel Collector pool -> field policy + bounded queue -> dedicated backend dataset Do not point every developer client directly at the primary observability backend. Give the collector write-only destination credentials, separate its dataset from production application telemetry, and define retention before rollout. Isolation is not anonymity. Stable user, device, organization, or repository identifiers may still be sensitive. Start with a field budget Category Initial policy Product and version Keep bounded values Operation and status Keep documented enums Timing and counts Keep numeric measures Raw prompts or generated code Drop by default File paths and repository URLs Drop or transform after review User identity Prefer scoped pseudonymous identity Free-form errors Drop raw text; keep reviewed classes These categories are recommendations, not a description of GitHub's payload. Inspect a restricted canary before naming actual keys. processors : memory_limiter : check_interval : 1s limit_mib : 512 spike_limit_mib : 128 attributes/field_budget : actions : # Illustrative keys only; replace after payload review. - key : user.email action : delete - key : file.path action : delete - key : command.arguments action : delete batch : send_batch_size : 512 timeout : 5s Verify processors against the chosen Collector distribution. A valid startup does not prove that records satisfy policy. Drill three failures Backend outage: block the exporter. Retries must be bounded, queue growth vi

This article is my submission for the Agents of SigNoz Hackathon: Blog Track, where participants...

TL;DR: If you only collect metrics, Prometheus Agent mode is lightweight, familiar, and difficult to beat. If you collect metrics, logs, or traces together, or expect to in the future, Grafana Alloy's unified pipeline is usually worth the additional complexity. Once you've decided to move from pull-based scraping to a push architecture , the next question is which agent should actually run on each host. In 2026, the two strongest choices are Prometheus Agent mode and Grafana Alloy. I run Alloy across my production fleet, but that doesn't automatically make it the right answer for everyone. The Shift in the Monitoring Landscape Over the last couple of years, Grafana has consolidated both metrics and log collection into Grafana Alloy. Grafana Agent reached end of life on November 1, 2025, and Promtail followed on March 2, 2026. Neither receives security fixes anymore. The practical choice moving forward: Feature Prometheus Agent Grafana Alloy Metrics ✅ ✅ Logs ❌ ✅ Traces ❌ ✅ Config Prometheus YAML Alloy components Footprint Smaller Larger Learning curve Low Moderate Future direction Metrics agent Unified telemetry The table gives the short answer. The rest of this article explains where those differences actually matter in practice. Prometheus Agent mode. Run the Prometheus binary with the --agent flag and it stops acting as a full Prometheus server. It no longer stores local TSDB blocks, evaluates alerting rules, or serves queries. Instead, it scrapes targets, buffers samples in a write-ahead log, and forwards them upstream via remote_write . It is Prometheus with the storage and query layers removed. Grafana Alloy. A single agent that collects metrics, logs, and traces, processes them in a component pipeline, and pushes each signal to its backend. It embeds many exporters directly, so a line like prometheus.exporter.unix "node_exporter" {} gives you full node_exporter functionality without installing a separate binary. The Case for Prometheus Agent If you only need m

How I shipped structured JSON logging + Prometheus metrics with zero new dependencies I almost added structlog and prometheus_client to my pyproject.toml . Then I read what they actually do. Both libraries are excellent. structlog is the right call when you have a 30-engineer team shipping 50 services. prometheus_client is the right call when you have five teams of consumers scraping different metrics. For a single-author Python project with one process and one user, both are over-engineered. The 80 lines of code I would have pulled in, I can write in 200. The result: zero new runtime dependencies, full control over the output, and a smaller pip install footprint for every user. Here is what I did instead. The minimum useful observability surface A small Python service needs four things, in order of importance: Every log line is one JSON object. (No parsing for downstream tools.) Every request has a trace id. Every log line in that request carries the same trace id. (So you can grep by id and see the whole story.) Every log line goes to stderr. (So journald , Docker, and kubectl logs all see it without any extra configuration.) Every metric is exposed in Prometheus text format at a stable URL. structlog gives you #1, #2, #3 with a lot of flexibility. prometheus_client gives you #4 with a lot of flexibility. Both are about 16 MB of transitive dependencies combined. For a service that runs in a single process and exports maybe 20 metric names, the libraries are doing more work than the project. The 80-line JsonFormatter The custom logging formatter is the simplest part. The whole thing is here: import json import logging from contextvars import ContextVar from datetime import datetime , timezone _trace_id_var : ContextVar [ str | None ] = ContextVar ( " trace_id " , default = None ) class JsonFormatter ( logging . Formatter ): def format ( self , record : logging . LogRecord ) -> str : payload = { " ts " : datetime . now ( tz = timezone . utc ). isoformat (), " level

Where Copilot's telemetry stream lands, decided centrally GitHub added a control on July 8 that lets an enterprise mandate where the Copilot Chat extension in VS Code and Copilot CLI send OpenTelemetry data, removing the need for individual developers to set OTEL_* environment variables. Per the GitHub changelog, the setting is delivered through a telemetry block in the enterprise-managed settings, and a managed value takes precedence over environment variables and user settings. Four things are configurable in the block: the OTLP export endpoint and transport ( otlp-http or otlp-grpc ), the OTel service name and resource attributes, exporter headers such as an authentication token for the collector, and whether prompt, response and tool content is captured, with a separate flag for whether developers can change that. Delivery uses the channels documented on the same page: native MDM (Windows Registry or macOS managed preferences), server-managed settings from a signed-in GitHub account, or a file-based managed-settings.json . Where this bites The precedence rule is the point. If a platform team owns the collector and needs traces routed to it, this is exactly the switch they wanted. If a developer had their own OTLP endpoint pointed at a local sink, they will see the session start emitting somewhere else. The changelog does not describe a per-user override once a managed value is set. A scoping note is worth reading twice. The changelog states that managed exporter headers apply only to the Copilot Chat extension's OTLP exporter. The endpoint and transport policy still reach the CLI agent host, but the auth-token flow the changelog calls out is bound to the Chat surface. On-call teams standing up the collector should plan for that asymmetry before it lands as a surprise during triage.

On March 3, 2026, Helicone announced it was joining Mintlify. If you run Helicone in production, the practical question is not whether the acquisition is good or bad. It is what changes for you, and whether you need to do anything about it. Here is the honest version, and a checklist if you decide to move. What actually changed Helicone's founders joined Mintlify, and active feature development on the standalone product has wound down. The team has said security patches, bug fixes, and new model support will continue. New features and roadmap work are the part that stopped. For a lot of teams that is fine for a while. A logging proxy that already works does not stop working the day the roadmap freezes. But two situations make people start looking. You are on Helicone Cloud and you want to know the plan is still moving forward, not just being kept alive. Or you self-host and you were counting on features that are now unlikely to ship. Helicone was one of three observability tools acquired in a few months. ClickHouse bought Langfuse and Cisco bought Galileo in the same window. If you are picking a replacement, that pattern is worth keeping in mind. More on that at the end. Do you even need to move right now Worth saying plainly. If you self-host Helicone, you are happy with it, and you do not need anything new from it, there is no fire. The code keeps running. You can migrate on your own schedule instead of someone else's. The case for moving sooner is stronger if you are on the hosted product, if you depend on the gateway staying current with new providers and models, or if you would rather switch once now than watch and decide later. If that is you, the rest of this is for you. The migration checklist Helicone and Spanlens are both drop in proxies, so the mechanical part is short. The work is mostly finding every place your code sets a base URL and updating headers. 1. Swap the base URL This is the one required change. // Before, Helicone const openai = new OpenAI (

Request logging: vì sao mỗi log line phải có request id, và structured log cứu debug thế nào khi có sự cố production Request logging không phải là in console.log('got a request') cho vui. Nó là dấu vết duy nhất còn lại khi một request đã đi qua service, đi qua vài downstream, gặp lỗi, và bị đóng socket. Nếu log không có gì để nối lại các dòng thuộc cùng một request — không có requestId , không có traceId — thì log của một service RPS trung bình biến thành một mớ text xen kẽ giữa hàng chục request đồng thời, và câu hỏi "request X đã đi tới đâu, fail ở service nào" trở thành không trả lời được. Structured log (mỗi dòng là một JSON object với field cố định) + một correlation id truyền qua toàn bộ pipeline chính là cái làm log có thể query được, thay vì grep mù. Hai framework Express và Fastify tiếp cận khác nhau: Fastify tích hợp pino sẵn và tự sinh req.id , Express phải tự dán qua pino-http hoặc middleware tay. Nhưng cả hai đều vỡ theo cùng một kiểu khi correlation id bị đứt giữa các service. Cơ chế hoạt động Ba miếng ghép: (1) một structured logger — thường là pino trong ecosystem Node vì nó ghi NDJSON và có async destination, (2) một cơ chế sinh/nhận requestId , (3) một cách propagate id đó qua async work và qua HTTP call sang service khác. pino là JSON logger tối giản: mỗi lời gọi logger.info({...}, 'message') xuất một dòng NDJSON ra một WritableStream (mặc định là stdout). Log level là số ( trace=10, debug=20, info=30, warn=40, error=50, fatal=60 , theo pino docs), và có child logger — logger.child({ reqId }) tạo một logger mới bind sẵn các field, mọi log line từ child đều có reqId mà không phải truyền tay. Với Fastify , chỉ cần logger: true là có pino và request logging tự động — Fastify sinh request.id (mặc định là monotonic counter, đổi được qua option genReqId ) và log một cặp dòng "incoming request" / "request completed" cho mỗi request. Trong handler, request.log là child logger đã bind reqId : import Fastify from ' fastify ' import crypto from ' node:crypto

The previous code-graph series was about reshaping a static analysis graph so AI could query it. The same kind of reshaping is needed on the observability side. This post walks through four axes — application / infrastructure / CI / LLM — and the deliberately different shapes each one ends up in. The design judgments worth calling out: computing Gemini cost client-side instead of from billing API, sending Claude Code OTel straight to BigQuery instead of Loki, and shipping CI logs via post-hoc pull instead of webhook push.

Book: Observability for LLM Applications — Tracing, Evals, and Shipping AI You Can Trust Also by me: Agents in Production — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub Your triage agent has been in production for three months. The traces look clean. Every span is green, every run terminates, the p95 latency is flat, the token bill is boring. Then support forwards you a screenshot: the agent routed a billing refund to the security queue. You pull the trajectory. Nothing is broken. The agent called a reasonable tool with reasonable arguments, got a reasonable result, and picked the wrong queue with total confidence. That is silent drift. The trace shows you what the agent did. It does not tell you whether what it did was any good. Between a model provider's minor version bump, a prompt tweak someone shipped on Tuesday, and the slow shift in your incoming traffic, the quality of your agent's decisions moves. It rarely announces itself with an error. It shows up as a support ticket, then a second one, then a churned account. You catch it the same way you catch a memory leak: with a baseline and an alarm, not by staring at dashboards. Decision quality is a distribution, not a number The failing traces in Chapter 12 of Agents in Production are the easy case. Twenty retries of the same empty search is visibly wrong. You see the loop count in the invoke_agent parent and you know. Most quality regressions are not visible in a single trace. They show up only when you look at the distribution of decisions across thousands of runs. So the first thing to instrument is the decision itself. If you followed the tracing chapter you already emit a small fixed vocabulary per chat span: span . set_attribute ( " gen_ai.agent.step " , 3 ) span . set_attribute ( " gen_ai.agent.decision " , " call_tool " ) span . set_attribute ( " gen_ai.

The Cloud Native Computing Foundation (CNCF) has announced the graduation of OpenTelemetry, elevating the project to the foundation's highest level of maturity and formally recognizing it as production-ready for enterprise use. By Craig Risi

Book: Decoupled PHP — Clean and Hexagonal Architecture for Applications That Outlive the Framework Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You already run three tools that half-cover this job. Pulse gives you a live wall on a local route. Datadog runs an agent and prices on host and usage volume, so the bill scales with your infrastructure. Sentry catches the exceptions after they already hurt someone. And none of them can tell you the one thing you actually asked: the checkout request that took 900ms at 14:03 dispatched a job, that job ran a query, and the query is what timed out. Laravel Nightwatch reached general availability in 2025 as the framework's own APM, aimed straight at that gap. It is worth knowing exactly what it captures, what it charges, and where its knowledge of your app stops and yours begins. What Nightwatch actually is Two moving parts. A Composer package inside your app, and a separate agent process that ships the data. composer require laravel/nightwatch The package writes events to a local socket. The agent listens on 127.0.0.1:2407 , batches what it receives, and sends it to Nightwatch's cloud. Because the agent runs outside your request cycle, the request thread is not blocked waiting on a network call to a telemetry backend. Laravel puts the added cost at under 3ms per request ; take that as a starting figure and measure your own before you trust it. # environment token per app + environment NIGHTWATCH_TOKEN = your-env-token # start the collector (keep it running under a # process monitor: Forge daemon, Vapor, supervisor) php artisan nightwatch:agent # confirm it is alive and receiving php artisan nightwatch:status One detail that bites people: the agent has to be running for anything to arrive. In local dev you start it by hand. In product

Instacart redesigned its personalized marketing system using a configuration-driven multi-tenant architecture on Storefront Pro. The system replaces retailer-specific implementations with a shared execution engine, enabling scalable personalization, faster configuration propagation in under a minute, and 99.9% delivery success across hundreds of retail banners through a unified campaign platform. By Leela Kumili

TL;DR — A Kafka + Flink + OTel ingestion pipeline cost us ~$700–800/month at 10 MB/s. We rebuilt it as a single binary where the data, the write-ahead log, and the Iceberg catalog all live in S3 alone — no Kafka, no local disks, no coordination service — for ~$100/month . Here's the design. Self-hosted observability sooner or later runs into the problem of storing state. Query load, CPU, and data volume can all be handled by scaling out, but the stateful layer is something you have to operate by hand. At first it's almost unnoticeable: a disk degrades here, replication falls behind there, a recovery hangs somewhere else. As the data grows, incidents stop being one-offs and start to recur. At some point your observability stack - whether it's Grafana Loki, Elastic, or ClickHouse - starts demanding the same attention as a full-blown database that you're on the hook for. Kubernetes operators cover some of these cases, but operating the state is still on you. Managed solutions take that burden away and bring their own: rising costs, ingestion-pipeline constraints, and limits on retention and cardinality. But if you'd rather not sign up for the constant operational grind - or live with the constraints of managed solutions - it's worth asking: can we take the stateful part out of operations entirely? Storage and format The first candidate for offloading storage responsibility is Amazon S3. S3 gives you what local disks can't: fault tolerance and practically unlimited scale, with no storage to manage yourself. It isn't free, though: data-access latency goes up, and you pick up separate costs for API operations. For OLTP workloads that's a dealbreaker. For observability workloads - which are dominated by sequential writes and analytical reads - these trade-offs are often acceptable. At first glance, this problem is already solved. Loki , for example, uses S3 as its primary storage. But according to Loki's public documentation (v3.6.x) at the time of writing, Loki doesn't re

Asymm Systems has released Eliya 25.0.3, an OpenJDK 25 LTS distribution aimed at improving production diagnostics in Java environments. It consolidates several HotSpot features into an opt-in Production profile. Eliya is designed for teams needing reliable diagnostic data, especially in regulated settings. Future enhancements are planned for Phase 2. By A N M Bazlur Rahman

When Logs and Metrics Aren't Enough You have great dashboards. Your log aggregation is solid. But when a user reports "the checkout page is slow," you still spend 30 minutes jumping between services trying to find the bottleneck. That's the gap distributed tracing fills. What Tracing Actually Shows You A trace is a complete picture of a single request as it flows through your system: User Request → API Gateway → Auth Service → Product Service → DB → Cache → Response 5ms 12ms 45ms 120ms 3ms ^ This is your bottleneck Without tracing, you'd see: API Gateway: latency looks fine Auth Service: latency looks fine Product Service: latency is HIGH but why? With tracing, you see the exact DB query inside Product Service that's taking 120ms. Getting Started with OpenTelemetry OpenTelemetry is the standard. Here's a minimal setup: # Python example with Flask from opentelemetry import trace from opentelemetry.instrumentation.flask import FlaskInstrumentor from opentelemetry.instrumentation.requests import RequestsInstrumentor from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter # Setup provider = TracerProvider () provider . add_span_processor ( BatchSpanProcessor ( OTLPSpanExporter ( endpoint = " http://otel-collector:4317 " )) ) trace . set_tracer_provider ( provider ) # Auto-instrument everything FlaskInstrumentor (). instrument_app ( app ) RequestsInstrumentor (). instrument () SQLAlchemyInstrumentor (). instrument ( engine = db . engine ) That's it. Three auto-instrumentations cover 80% of what you need. Custom Spans for the Other 20% Auto-instrumentation gives you HTTP calls and DB queries. Add custom spans for business logic: tracer = trace . get_tracer ( __name__ ) def process_order ( order ): with tracer . start_as_current_span ( " process_order " ) as sp

We moved off our first tracer in month eight. The migration took one engineer the better part of a sprint, because the trace data lived in a schema we did not own. Nobody costed that line item on day one. I am writing this so you can. I run reliability for a small team shipping LLM features. When the pager goes off at 2am, I do not care which dashboard is prettiest. I care about two numbers: what this tool costs me per month, and what it costs me to leave. Those two numbers are the whole story, and they are almost never on the comparison page. So here are six Langfuse alternatives. For each I tracked both numbers: the monthly bill on the invoice, and the exit bill that only shows up the day you migrate. I compared Helicone, Arize Phoenix, LangSmith, Braintrust, Laminar, and Future AGI traceAI. They all trace LLM calls (prompts, tokens, retrieval spans, latency). The axis that decides your exit cost is whether the trace format is OpenTelemetry-native or a vendor schema. Get that wrong and the migration bill lands later, with interest. The cost nobody puts on the pricing page Your monthly invoice is the visible cost. The exit cost is the invisible one: re-instrumenting the app, rebuilding integrations, and losing historical traces when the schema does not travel. If your spans are OTel, the exit cost trends toward zero because the data is portable by construction. If they are proprietary, you are paying a deferred bill every month you stay. Sort on that first. Helicone. The gateway-first option. You proxy model calls through it and get logging, cost tracking, and analytics with almost no code change. Apache-2.0, self-hostable, roughly 5,800 GitHub stars as of June 2026. On pure observability ergonomics this is one of the strongest picks, and the proxy model means low setup cost. The thing to watch at scale: a gateway in the request path is one more hop to reason about when latency spikes. Arize Phoenix. The open-source OTel option. Tracing plus evals, self-hostable, a

Zabbix is an open-source monitoring platform that tracks the health and performance of servers, networks, applications, and services, with built-in alerting and visualisation. This guide deploys the Zabbix server, web UI, agent, and MySQL database using Docker Compose, with Traefik handling automatic HTTPS for the dashboard. By the end, you'll have Zabbix monitoring its own host with a secured dashboard at your domain. Set Up the Project Directory 1. Create the project directory: $ mkdir ~/zabbix-docker $ cd ~/zabbix-docker 2. Create the environment file: $ nano .env DOMAIN = zabbix.example.com LETSENCRYPT_EMAIL = admin@example.com MYSQL_PASSWORD = YOUR_DB_PASSWORD MYSQL_ROOT_PASSWORD = YOUR_ROOT_PASSWORD Deploy with Docker Compose 1. Add your user to the Docker group: $ sudo usermod -aG docker $USER $ newgrp docker 2. Create the Compose manifest: $ nano docker-compose.yaml services : traefik : image : traefik:v3.6 container_name : traefik restart : unless-stopped command : - " --providers.docker=true" - " --providers.docker.exposedbydefault=false" - " --entrypoints.web.address=:80" - " --entrypoints.websecure.address=:443" - " --entrypoints.web.http.redirections.entrypoint.to=websecure" - " --entrypoints.web.http.redirections.entrypoint.scheme=https" - " --certificatesresolvers.le.acme.httpchallenge=true" - " --certificatesresolvers.le.acme.httpchallenge.entrypoint=web" - " --certificatesresolvers.le.acme.email=${LETSENCRYPT_EMAIL}" - " --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json" ports : - " 80:80" - " 443:443" volumes : - /var/run/docker.sock:/var/run/docker.sock:ro - ./letsencrypt:/letsencrypt networks : - zabbix-net mysql-server : image : mysql:8.4.8 container_name : zabbix-mysql environment : MYSQL_DATABASE : zabbix MYSQL_USER : zabbix MYSQL_PASSWORD : ${MYSQL_PASSWORD} MYSQL_ROOT_PASSWORD : ${MYSQL_ROOT_PASSWORD} volumes : - ./mysql-data:/var/lib/mysql networks : - zabbix-net restart : unless-stopped zabbix-server : image : zabbix/zabbix-se

Good architecture is not only about how a system is built. It is also about how well the team can understand that system once it is running. That is where observability belongs in the architecture conversation. It is common for observability to be treated as something that comes after the main engineering work. The service gets built. The API works. The deployment succeeds. Then, somewhere near the end, the team starts thinking about logs, dashboards, alerts, and operational visibility. That approach creates a gap. The architecture may look clean on paper, but once the system is in production, the team has to understand how it behaves under real conditions. Real users do not follow the happy path perfectly. Dependencies slow down. Queues back up. Data arrives in unexpected shapes. Deployments change behavior in ways that are not always obvious. If the system does not give the team a way to see those things clearly, the architecture is incomplete. Observability is not decoration around the system. It is part of the system design. Architecture Describes the System. Observability Shows the Truth. Architecture is built on assumptions. During design, teams make reasonable guesses about usage patterns, service boundaries, dependency behavior, data flow, scale, latency, and failure modes. Some of those assumptions are based on experience. Some are based on current requirements. Some are simply the best call the team can make with the information available at the time. That is normal. The problem is not that architecture contains assumptions. Every architecture does. The problem is when those assumptions cannot be tested once the system is real. A design might assume that an external dependency will be reliable enough. Production may show that it is the slowest part of the request path. A queue might look like a clean decoupling point during design. Production may reveal retry behavior, duplication, or ordering concerns that were not obvious upfront. A serverless function m

Dan Fineran explores how eBPF has evolved far beyond its roots in packet filtering into a robust, safe way to extend the Linux kernel. He explains how the eBPF "verifier", the security guardrail, enables implementation of deep observability and networking without the risks of traditional kernel modules or the slow upstreaming process. By Dan Fineran