Why I stopped guessing at Spark and dbt config values

I've spent more than a decade building data pipelines, and the part nobody warns you about isn't the pipeline logic. It's the tuning. Executor memory, shuffle partitions, cluster size, thread counts. You pick numbers, ship it, and a few weeks later something breaks in a way that's obviously tuning-related but not obviously what to change . The pattern repeats enough times that you start recognizing it before you've even opened the logs. Job's slow, thousands of tiny shuffle tasks, someone way overestimated the partition count. Job dies on OOM, memory's set for last quarter's data volume, nobody updated it since. Cloud bill jumps, a cluster's been sized for peak load and just sits there mostly idle the other 20 hours a day. Every senior data engineer has this pattern-matching running in their head. It's tribal knowledge, and it lives in one or two people's heads on most teams, which means it doesn't scale and it definitely doesn't survive someone leaving. So I built a small tool to make that pattern-matching explicit instead of tribal: it reads your pipeline's config alongside its actual run metrics and tells you what's likely wrong, with the reasoning shown, not just a suggested number. Why rules instead of a model The obvious move in 2026 is to reach for an ML model. I didn't, and it wasn't because I don't think ML has a place here eventually. It's that for this specific problem, a handful of threshold rules already gets you most of the value, and they're something you can actually audit. If a rule fires, I can point at the exact condition and the exact number: average heap usage 28%, peak 47%, five runs, no OOM errors, therefore memory's over-provisioned, shrink it by roughly a fifth. That's checkable. You can look at your own metrics and see whether the reasoning holds. A model's confidence score doesn't give you that, and for something that's about to change a production config, I want the person approving it to be able to say "yes, I see why" rather than "the m