A/B Test Lab (Binary)

Use this after your test has run and you have results for Variant A and B.

This answers: Is the observed difference likely real? And how big could it actually be? You’ll see both a statistical significance result and a confidence interval showing the plausible range of the true effect.

The confidence interval matters because it shows uncertainty. A result can be “statistically significant” but still too small to matter for your business. Always interpret the size and range of the effect — not just whether it crosses a threshold.

Not for: multi-variant tests, continuous metrics (e.g., revenue per user), or stopping a test early without a plan.

Use this before you run a test.

This answers: How much traffic do I need to reliably detect a meaningful difference? It helps you estimate required sample size based on your current baseline and the minimum detectable effect (MDE).

MDE means the smallest change you would actually care about. If you want to detect very small lifts, you’ll need much larger samples. Small effects require large traffic — there’s no shortcut.

Not for: analyzing results after the fact. If you underpower a test and get a null result, this tool won’t “rescue” it.

Use this if you plan to check results multiple times during a test.

This answers: How can I look at the data without inflating false positives? When you check daily and stop as soon as something looks significant, you increase the chance of being wrong. This tool adjusts decision thresholds based on planned looks.

It supports structured, pre-defined checks — not arbitrary continuous monitoring. If you refresh the dashboard every hour and stop whenever you feel confident, no statistical adjustment can protect you.

Plan your looks in advance. Then follow the plan.

Use this as soon as traffic starts flowing.

This answers: Is my experiment split behaving as expected? Sample Ratio Mismatch (SRM) happens when the actual traffic split deviates meaningfully from what you intended (e.g., 50/50 becomes 60/40).

SRM often indicates tracking bugs, targeting issues, bot traffic, or broken randomization. If SRM is present, your experiment may be invalid — even if results look significant.

Do not interpret test results before ruling out SRM. Fix allocation issues first. Then restart or rerun the test if necessary.