A/B Test Lab (Binary)
I built this A/B testing lab for binary outcomes like conversion rate, signup rate, and CTR.
Use it when you want disciplined decisions from test data without pretending uncertainty does not exist.
Decision canvas
Run calculations below. Confidence, practical impact, and recommendation stay pinned here while you work.
SRM status
Analysis workspace
Use the tabs for significance, power planning, sequential thresholds, and SRM diagnostics.
Significance + Effect
Use this after your test has run and you have results for Variant A and B.
This answers: Is the observed difference likely real? And how big could it actually be? You’ll see both a statistical significance result and a confidence interval showing the plausible range of the true effect.
The confidence interval matters because it shows uncertainty. A result can be “statistically significant” but still too small to matter for your business. Always interpret the size and range of the effect — not just whether it crosses a threshold.
Not for: multi-variant tests, continuous metrics (e.g., revenue per user), or stopping a test early without a plan.
Use independent binary outcomes with fixed assignment.