- What statistical test does this use?
- A two-proportion z-test with pooled standard error under the null hypothesis. This is the textbook frequentist test for comparing two conversion rates and the default p-value reported by Optimizely, VWO, Google Optimize, Statsig, and most internal experimentation platforms. The formula: z = (p̂_v − p̂_c) / √(p̂(1−p̂)(1/n_c + 1/n_v)), where p̂ is the pooled rate. The p-value is two-tailed: p = 2(1 − Φ(|z|)). Two-tailed is the right default because you usually care about a difference in either direction (the variant might be worse than the control, not just better).
- What does 'statistically significant' actually mean here?
- It means p < 0.05 — under the null hypothesis that the two arms have the same true conversion rate, you'd see a difference at least this large by random chance less than 5% of the time. That's the conventional threshold; it's a choice, not a law of nature. A 'significant' result still has a ~5% false-positive rate, which is why you don't read one p-value and ship: you decide the sample size in advance, run to that size, then read the result once.
- Why is my p-value greater than 0.05 even though the variant looks better?
- Three usual reasons. (1) Sample size is too small — small experiments can't detect small lifts no matter how real the lift is. (2) The actual effect is smaller than it looks at this sample size — what looks like a 'clear 8% lift' on 500 visitors per arm is well within the random noise of two coins flipped 500 times each. (3) There genuinely isn't an effect. Use the sample-size planner to figure out how many visitors you'd need to detect the lift you're seeing; if your test is underpowered, either run it longer or accept that you can't measure effects this small at this traffic.
- What's the difference between relative lift and absolute lift?
- Absolute lift is the difference in conversion rates, in percentage points (pp): 12% − 10% = 2pp. Relative lift is the percent change: (12% − 10%) / 10% = 20%. Most product teams discuss lift in relative terms ('the variant lifted conversion by 20%') because that's how the business impact scales. But the math of significance testing uses the absolute difference. The calculator reports both.
- How big does my sample size need to be?
- It depends on your baseline rate, the smallest lift you want to detect (MDE), your significance threshold α, and your statistical power. As a rough guide: detecting a 10% relative lift on a 5% baseline rate at α = 0.05 with 80% power requires about 31,000 visitors per arm — 62,000 total. The rule of thumb: sample size scales as the inverse square of the effect size. Halving the MDE quadruples the required visitors. Use the Sample-size planner tab to compute the exact number for your scenario.
- What's a 95% confidence interval and how do I read it?
- It's the range that contains the true conversion rate 95% of the time across repeated experiments. If the variant's CI is [11.4%, 12.6%] and the control's CI is [9.4%, 10.6%], the two intervals don't overlap, which is consistent with a real difference. If the intervals overlap a lot, the result is probably not significant. CIs and p-values are two views of the same statistical information — the calculator shows both because different audiences prefer different framings.
- Why shouldn't I peek at p-values mid-test?
- Because every time you check, you give the test another chance to cross the 0.05 line by random noise. Checking 10 times during a test inflates your false-positive rate from 5% to about 20%. This is called 'optional stopping' or 'p-hacking,' and it's the most common way real teams ship 'wins' that don't replicate. Two ways to do it right: (1) decide the sample size up front, run to it, read once. (2) Use a sequential testing method (Bayesian, group-sequential, always-valid p-values) that's designed for repeated peeking — but those aren't what this calculator implements.
- What's statistical power and why does it default to 0.80?
- Power (1 − β) is the probability that your test correctly detects a real effect — the inverse of the false-negative rate. 80% power means: if the variant really is better by your MDE, you'll catch it 80% of the time. 0.80 is the industry default — high enough that you usually find real wins, low enough that the required sample size is achievable. Drug trials often use 0.90; quick growth experiments sometimes drop to 0.70. Raising power costs more visitors.
- What if my variant is worse than my control?
- The math works in both directions. The calculator reports the winner as control when its rate is higher, variant when variant's rate is higher, tie when they're equal. The two-tailed p-value asks 'how different are they?' regardless of direction — so a control that's significantly better than the variant gets a small p-value too. Practically, a significantly-worse variant is just as actionable as a significantly-better one: ship the control, kill the variant.
- Should I use a one-tailed or two-tailed test?
- Two-tailed, almost always. One-tailed only makes sense when you've pre-committed to a direction before seeing any data, AND you genuinely don't care about an effect in the other direction. In practice this is rare — you usually care whether the variant did the opposite of what you expected. The two-tailed test is the conservative, defensible default; this calculator uses it.