Three ways my grouped train/test split leaked anyway...

I spent two weeks building a computer vision component to estimate how full a plastic container is from drone imagery. Translucent white containers, whitish chemical product inside, shot obliquely from a drone during field inspections. The headline number looked good: mean absolute error of 0.055 on fill fraction, Pearson correlation of 0.97. Then I audited my own evaluation and found that 38 of my 46 test crops had the same physical container sitting in the training set. The arithmetic was fine. The problem was the sentence I had wrapped around it: I was presenting 0.055 as the error on containers the model had never seen before. What makes this worth writing about is that I had the guardrail in place from day one, and it failed three separate times for three unrelated reasons. Each one is easy to reproduce in any project that trains on frames extracted from video. Why grouping matters here at all A drone flies over a site and captures a burst. In my case, 12 frames over 12 seconds. The same physical container appears in every frame of that burst, from slightly different angles and distances. If you shuffle those crops randomly into train and test, you are asking the model to recognize a container it has already memorized. The metric you get back describes interpolation between frames of one burst. It says nothing about a container the model has never seen. This is the most common failure in applied ML and everyone knows about it. Which is exactly why the next part is worth reading. The guardrail I wrote on day one My dataset module reads the grouping column from config and does not offer a random option at all: split : group_column : skid_id # never random The code path for a random split does not exist. You cannot pass a flag to get one. I wrote it that way on purpose, on the first day, before there was any data to split. I still leaked. Three times. Leak 1: the group column held the wrong ID group_column was set to skid_id , which is what you want. Group by phys