I analyzed 31,352 hourly LLM benchmark scores: within-day variation was 2.8 points, while between-day variation was 8.4 [P]

https://preview.redd.it/42s57e5oqamh1.png?width=1903&format=png&auto=webp&s=69958a72e22276534b3605d11f3e1721f76e59c9 Disclosure: I developed AIStupidLevel, the open-source system used to collect and analyze this data. Both the frontend and backend are MIT-licensed. Most LLM evaluations measure performance at a single point in time. I wanted to investigate a different question: How stable are the models behind production APIs over time, and how can sustained performance changes be separated from ordinary stochastic variation? I built a continuous evaluation pipeline that repeatedly tests models across coding, deep reasoning, tool calling and high-frequency canary tasks. Dataset and evaluation process For this analysis, I examined: 31,352 hourly benchmark scores 49 model identifiers Multiple providers and model families Repeated measurements using consistent tasks and scoring A normalized 0-100 composite score Coding responses are executed rather than judged only through model-based evaluation. Tool-calling tests require models to select tools, construct valid arguments and complete workflows inside isolated Docker environments. Tasks are executed five times, with the results aggregated to reduce the influence of unusually strong or weak generations. Prompts, scoring logic and API parameters are kept consistent wherever supported by the provider. Main result The observed score variation was: Within the same day: 2.8 points Between different days: 8.4 points Between-day variation was therefore approximately 3× greater than within-day variation. This suggests that isolated hourly movements are dominated by normal model stochasticity, while sustained changes across daily evaluation windows provide a materially stronger signal for detecting performance drift. The detection pipeline aggregates repeated measurements into daily medians and applies sequential change-point detection. Potential incidents must persist beyond the expected historical variance and pass statistical