Your A/B test has three goals and they disagree. Now what?

Every A/B testing tutorial ends the same way: run the test, wait for significance, ship the winner. Then you run a real test and variant B converts 12% better on newsletter signups, brings in 4% less revenue per visitor, and bounce is flat. Nothing is significant except the signups. Ship it? I spent an embarrassing amount of time on this question while building an A/B engine, and most of what I read online didn't help, because most of it assumes one metric. This post is what I ended up with. It's not novel — the statistics are decades old — but I couldn't find it written down in one place with working code, so here it is. Why the p-value doesn't answer the question you're asking Two problems, and the second one is the bad one. Multiple comparisons. Three metrics at α = 0.05 means roughly a 14% chance of at least one false positive if nothing is actually different. Bonferroni fixes this, but now you need α = 0.017 per metric and your test needs to run three times as long. On a site doing 300 conversions a month that's not a fix, it's a refusal. The p-value is answering a different question. It tells you the probability of your data assuming no difference exists. What you actually want to know is: if I ship B, how much do I expect to lose if I'm wrong? Those are not the same question and no amount of Bonferroni turns one into the other. There's also the peeking problem — everyone checks the dashboard daily and stops when it goes green, which quietly inflates the false positive rate well past whatever α you wrote down. I'll come back to that, because Bayesian methods do not magically solve it, whatever you may have read. Posterior first, decision second For a conversion rate, the Beta-Binomial conjugate pair gives you the posterior in one line. With a uniform prior, after c conversions out of n visitors: p | data ~ Beta ( 1 + c , 1 + n - c ) That's it. No closed-form comparison between two Betas that's worth implementing, so sample. PHP has no Beta sampler in core, and