标签：#trace

Originally published at https://blog.pathvector.dev/protocol-lab-trace-19/ — part of the free Protocol Lab series. This post is part of Protocol Lab , a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab . Every IP packet carries a TTL (time to live) that each router decrements by one. When it reaches zero, the router drops the packet and sends back an ICMP time-exceeded message. traceroute turns this rule into a map: send probes with TTL 1, 2, 3, … and each dying probe reveals the router at that distance. Reading guide: rfc-notes/traceroute-ttl.md Prerequisite: TCP Lab 07: Handshake and Teardown (reading captures) Expected time: 40–55 minutes. The Goal This lab builds a real multi-hop path and shows the mechanism: client → r1 → r2 → server , with two Linux routers in the middle, traceroute from the client lists each hop: 10.0.1.2 (r1), 10.0.2.2 (r2), 10.0.3.2 (server), a packet capture shows the ICMP time-exceeded replies (from r1 for TTL 1, from r2 for TTL 2) that traceroute is built on. By the end, you should be able to explain this table: Probe TTL Dies at Reply 1 r1 ( 10.0.1.2 ) ICMP time-exceeded from r1 2 r2 ( 10.0.2.2 ) ICMP time-exceeded from r2 3 server ( 10.0.3.2 ) reaches the destination What You Will Learn What the IP TTL field is for (loop protection) and how routers decrement it. What an ICMP time-exceeded message is and who sends it. How traceroute uses increasing TTLs to discover each hop. Why the hops appear in order, and why the last hop is the destination itself. The difference between forwarding (routers) and being an endpoint. This lab does not cover: UDP vs ICMP vs TCP traceroute probe types in depth (we use ICMP mode). Load-balanced paths (ECMP) where hops can vary between probes. Why some hops show * * * (rate limiting or filtered ICMP) in the real internet. Where to Rea

Why Workflows Need a Dedicated Evaluation Framework Traditional software testing covers code correctness. Workflows add two layers of uncertainty: LLM output is non-deterministic : the same input can produce different results across runs Cross-step dependencies : a Phase 3 problem may only surface at Phase 7, making the debugging chain long Without an evaluation framework, every workflow change requires a full end-to-end run: slow, expensive, incomplete coverage. Three-layer testing decomposes the problem. Three-Layer Evaluation Structure Layer 3: End-to-end tests (Workflow level) Full pipeline from trigger to completion Test cases: eval/cases.yaml Metrics: completion rate, Phase 4 avg rounds, gate trigger rate Layer 2: Integration tests (Phase level) Cross-step data flow is correctly passed Cross-phase routing logic fires correctly Layer 1: Unit tests (Step level) Each subagent's output matches its output contract No real LLM calls — validates JSON schema only Test priority: Layer 1 should be the most numerous and fastest — catches contract violations in seconds. Layer 3 is the slowest and most expensive — run it only when changes affect the main pipeline. Layer 1: Step-Level Unit Tests Unit tests verify that subagent output files match the declared schema. No real LLM calls needed. # tests/unit/test_phase3_output.py import json from pathlib import Path def test_analysis_output_schema (): """ Phase 3 output must conform to analysis_final.json schema """ output = json . loads ( Path ( " test_fixtures/phase3/analysis_final.json " ). read_text ()) assert " passed " in output assert isinstance ( output [ " passed " ], bool ) assert " confidence " in output assert 0.0 <= output [ " confidence " ] <= 1.0 assert " root_cause " in output assert isinstance ( output [ " root_cause " ], str | type ( None )) assert " evidence " in output assert isinstance ( output [ " evidence " ], list ) # on failure, error field must be present and non-empty if not output [ " passed " ]: ass

TL;DR We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Dynatrace via the Log Ingest API v2. The real value: Dynatrace's Davis AI can automatically correlate file access anomalies with application performance degradation — answering "why is the app slow?" with "because 500 users hit the same NFS share simultaneously." FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Dynatrace Log Ingest API v2 │ ▼ Davis AI ┌───────────────────┐ │ Correlates: │ │ • File access │ │ anomalies │ │ • APM metrics │ │ • Infrastructure │ │ health │ │ │ │ → Root cause │ │ in seconds │ └───────────────────┘ Verified on Dynatrace SaaS Trial (Tokyo-equivalent region). Logs visible in Logs Viewer within 1-2 minutes. This is Part 11 of the Serverless Observability for FSx for ONTAP series. Why Dynatrace for FSx for ONTAP? Most observability tools treat storage logs as isolated data. Dynatrace is different — it builds a topology map of your entire stack and uses Davis AI to find causal relationships through time-window correlation and entity connectivity: Scenario Without Dynatrace With Dynatrace App latency spike "Check the logs" Davis AI detects temporal correlation: file access to /vol/data/ increased 10x within the same 5-minute window as app response time degradation, connected via topology (app → NFS mount → SVM) Storage I/O anomaly Manual investigation Automatic correlation via shared topology entities — Davis identifies which services are affected based on entity relationships User reports slow file access Grep through audit logs DQL query + topology view showing the full dependency path from user request to storage operation The key differentiator: Davis AI correlates events across entities that share topology connections within overlapping time windows — not just keyword matching or manual dashboard correlation. Architecture ┌─────────────────────────────────────────────────────────┐ │ Event Sources │ ├─────────────────────────────────────────