What are the most common A/B test mistakes and how do you detect them?
Anyone on an experimentation team gets this. Listing "peeking" isn't enough — you need to know how to detect and mitigate each.
Peeking: stopping when p < 0.05 inflates false positives. Fix: pre-register sample size, or use sequential tests that account for repeated looks.
Sample Ratio Mismatch (SRM): traffic allocated 50/50 but observed split ≠ expected — assignment bug or logging bug. Chi-square on counts, reject any experiment with p < 0.001 on SRM.
Novelty/primacy effect: users behave differently when they see a new UI — effect decays or grows over time. Run longer, segment by cohort.
Multiple comparisons: testing 10 metrics at α=0.05 → ~40% chance of a false positive somewhere. Bonferroni or Benjamini-Hochberg correction.
External validity: Tuesday 10am ≠ weekend evening. Run for at least one full week, ideally two.
First thing I check on any A/B result is SRM — did the split actually land where we allocated? A chi-square on event counts; if p < 0.001, I don't trust the experiment, something leaked or got mis-bucketed. Next is peeking: I want to see the sample-size decision pre-registered, and if stakeholders peeked and stopped early, I treat results as exploratory. Novelty effect I catch by segmenting time: if week 2 lift is half of week 1, the shiny-new factor drove most of it. For multiple comparisons I apply Benjamini-Hochberg — more powerful than Bonferroni when many hypotheses are tested. External validity: always at least one full week because weekday/weekend composition matters. The most common real disaster I've seen: someone celebrates a 5% lift, but two months later it's gone — classic novelty plus regression-to-mean.
Always define guardrail metrics (revenue, latency, error rate) to catch "feature works but breaks other things".
Power analysis before launch — you need N = (Z_α+Z_β)² × 2p(1-p) / δ² for baseline rate p and MDE δ.
HireMe AI (即答侠) drills each pitfall with a detection recipe so interviewers see systematic thinking.
For large samples (CLT), yes. For small or very low rates, chi-square or z-test for proportions is more correct.
Minimum: full weekly cycle + pre-calculated sample size. Shorter = weekday bias + underpowered.
HireMe AI listens to your interview in real-time, detects questions, and generates instant answer suggestions — so you never freeze up again.
Try HireMe AI Free