How to tell an ad experiment is unwinnable before you run it
Most experiments that come back "no clear winner" were unwinnable on the day they launched. The data could not resolve an effect that size, and no amount of extra runtime was going to change that. You can find this out in about two minutes, before you spend anything, with one formula and a resampling pass over your own data. Here is the check, in three steps. Step 1. Compute the smallest lift your data can see For a two-arm test on a conversion rate, the smallest lift detectable at 95% confidence and 80% power is a one-liner: from math import sqrt Z_ALPHA = 1.96 # two-sided 95% Z_BETA = 0.84 # 80% power def mde ( baseline_cvr : float , n_per_arm : int ) -> tuple [ float , float ]: """ Minimum detectable effect: absolute (pp) and relative (%). """ se = sqrt ( 2 * baseline_cvr * ( 1 - baseline_cvr ) / n_per_arm ) abs_lift = ( Z_ALPHA + Z_BETA ) * se return abs_lift * 100 , abs_lift / baseline_cvr * 100 At a 3% conversion rate: clicks per arm smallest lift you can detect 5,000 +32% relative 20,000 +16% relative 100,000 +7% relative Read the middle row twice. Twenty thousand clicks per arm is a serious amount of traffic for a mid-market account, and a real 15% improvement still lands inside the confidence interval. The report will say "inconclusive," and the team will read that as a verdict on the idea. It is a verdict on the instrument. Invert the same formula and the planning question gets easier: at 3% baseline, detecting a 10% lift needs about 51,000 clicks per arm, and detecting a 5% lift needs about 203,000. If your account produces 8,000 clicks a month, you now know the honest answer to "how long should we run this." Step 2. Stop assuming your conversions are independent The formula above treats every click as an independent coin flip with the same probability. Account data does not behave that way, and the gap is not small. In a corpus of 31 advertiser accounts I maintain for diagnostic work (9.46 million search term rows, roughly $133M of spend, September 2024