Workshop: Gate Retrieved Context With a Cheap Scoring Pass in 70 Minutes

Untrusted retrieval is now a more common production failure than a weak prompt, because agents ingest memory they never score. A seventy-minute workshop can add a cheap scoring gate, a replayable log, and a reject path before generation. Students leave with a runnable Python harness, a four-row decision table, and a timing plan they can repeat. The method stays useful if every product name is removed and the scoring host is only a free server. What you will build This workshop treats retrieved snippets as untrusted input, not as ground truth the model should quote. You will capture a retrieval batch, score each chunk against a written rubric, and allow only passing chunks into the prompt. A JSONL replay log records the fingerprint, score, and decision so later failures can be diffed. The generation model never sees dropped text, which keeps stale or planted memory out of the answer. Timing box 00:00–00:10 — install dependencies, copy the harness, and load the sample corpus 00:10–00:30 — Exercise 1: capture retrieval payloads and stable fingerprints 00:30–00:50 — Exercise 2: score chunks with a rubric and an optional free model 00:50–00:65 — Exercise 3: gate the prompt and replay one rejected case 00:65–00:70 — debrief against the decision table and list remaining holes The schedule is a teaching box, not a production SLA, and it assumes one laptop plus one HTTP scoring endpoint. If the endpoint is slow, freeze Exercise 2 after five scored chunks and continue with the logged samples. Do not expand the window to chase a perfect judge; the learning goal is a gate you can rerun. Why a scoring pass belongs in front of generation Cheap code generation has made it easy to wire a retriever into a chat loop in an afternoon. The failure mode that follows is quieter than a crash: the model answers fluently from a chunk that is expired, off-topic, or injected. Architecture diagrams rarely show that hop as a trust boundary, so teams skip scoring and jump to a larger generator. A