Measuring LLM performance drift: observations and methodology from 31,352 repeated benchmark measurements [D]

One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots. A model is evaluated, a score is published, and we tend to talk about that score as if it describes a relatively stable object. But with API-served models, the thing behind the model name can change over time: serving infrastructure changes, provider configurations change, versions change, and sometimes behaviour changes without an obvious public version transition. So we started approaching benchmarking as a longitudinal measurement problem rather than a leaderboard problem. We continuously evaluate models across coding, multi-turn reasoning and tool use, while also running lightweight probes at a higher frequency. The important part for us is not simply asking "which model scores highest?", but: Is the model behaving differently from its own previous baseline? Is the change larger than its normal repeated-call variability? Did the benchmark configuration itself change? Is the effect concentrated in a particular task? Is it correlated across models from the same provider? Is an apparent degradation actually an availability/infrastructure issue rather than a capability change? One historical analysis covered 31,352 repeated score observations across 49 models . The standard deviation of within-day scores was 2.80 points , while the standard deviation of between-day daily medians was 8.43 points . That is roughly a 3:1 difference. I don't think this result by itself establishes that providers are changing models day-to-day - there are too many possible confounders for that conclusion. Task composition, sampling, missingness, provider behaviour and methodology changes all matter. But it was enough to convince us that temporal variation deserves to be measured rather than treated as noise around a permanent leaderboard score. Our current approach therefore keeps benchmark configurations versioned and only compares longitudinal observations produced under comp