标签：#reinforcementlearning

Recap. Part 1 framed the supervision problem. Part 2 architected the four-model system on AWS and counted the hardware. Part 3 put the gate on the page in fifteen lines of PyTorch. Now the question that decides whether any of it mattered: does the gate actually earn its keep - and what would it cost to know? This is the close. A verification design, the stability instrumentation that separates a real evaluation from a lucky one, and the FinOps reality that is the reason this whole series is a blueprint instead of a benchmark. SDAR makes two claims, not one It's tempting to reduce the paper to a single number. The reported gains over plain GRPO are real and worth stating - their numbers, not mine : roughly +9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop accuracy. But the headline percentage is the less interesting claim. The one that matters is the second: SDAR avoids the training instability that naive GRPO+OPSD falls into . The whole point of the gate Part 3 was to keep the teacher's noisy rejections from destabilizing training. So a verification that only measures final task success has tested half the paper. You have to measure stability too - and most reproductions don't. That two-pronged claim dictates the experiment. The verification: three arms, not two You cannot prove SDAR's thesis with a before/after. You need three training runs, identical except for the supervision: Arm What it is What it proves A. GRPO Plain agentic RL, no teacher The baseline the +9.4% is measured against B. Naive GRPO+OPSD Teacher distillation, no gate The instability SDAR claims to fix C. SDAR Gated distillation Part 3 Beats A on score and beats B on stability Drop arm B and you can show SDAR beats GRPO, but you've quietly deleted the paper's actual contribution - there's no evidence the gate did anything a plain teacher wouldn't. Arm B is the expensive arm nobody wants to run and the one that makes the result credible. Metrics that actually test the claim Metric What it mea

About this series. I'm going to take a fresh paper - Self-Distilled Agentic Reinforcement Learning (SDAR, arXiv:2605.15155 ) - and architect it end to end on AWS: the system design, the actual gate code, the evaluation plan, and a brutally honest cost model. What I'm not going to do is wave a benchmark number around. Reproducing a paper like this costs thousands in GPU time, and I'd rather show you the machinery than a screenshot you can't audit. The design is the deliverable. This is Part 1. A small, infuriating problem Picture an LLM agent working a web-shopping task. It reads the goal, searches, clicks a category, filters, opens a product, compares, adds to cart - twelve steps in all. At the end, it bought the wrong thing. So you do what reinforcement learning tells you to do: you score the trajectory. Reward = 0. Bad agent. Now answer this: which of the twelve steps was actually wrong? Maybe step 3, the search query, was fine and step 9, a filter choice, doomed everything. Maybe steps 1–11 were brilliant and step 12 fat-fingered the wrong button. Your single scalar reward has no idea. It punishes all twelve equally, including the eight that were correct. That's the supervision problem in agentic RL, and it's the thing this whole series is about. Why "just use RL" isn't enough for agents RL has become the default way to post-train LLM agents. The catch is that the reward usually lands at the trajectory level - one number for the entire multi-step episode. For a single-turn task ("answer this question"), that's tolerable; the action and the outcome are close together. For a long-horizon agent - ten, twenty, fifty turns of searching, calling tools, and reacting to an environment - it's a disaster of credit assignment . The signal is too coarse to tell the model which decisions earned the reward and which torpedoed it. You can throw more episodes at it and let statistics sort the credit out eventually. But "eventually" on a 30-turn task burns a lot of expensive comp