"A product team at ByteDance ran an A/B test and claims the new feature increased engagement by 5% with p < 0.05. As a data analyst, what questions would you ask before accepting this conclusion?"
ByteDance runs thousands of A/B tests, and data analysts serve as the gatekeepers of experimental rigor. This question tests your understanding of common statistical pitfalls in experimentation, your ability to distinguish valid from misleading results, and your communication skills in challenging conclusions diplomatically.
Check the experiment design: was the sample size pre-calculated, was the test duration pre-determined, and was the primary metric pre-registered?
Investigate potential biases: sample ratio mismatch, novelty/primacy effects, and selection bias in the treatment population.
Evaluate the statistical methodology: was multiple testing corrected for, was the test peeked at before completion, and is the metric normally distributed or does it require non-parametric methods?
Assess practical significance: is the 5% lift meaningful for the business, what is the confidence interval width, and does it account for long-term effects?
I would systematically validate the result across four dimensions. First, experiment integrity: check for sample ratio mismatch—if the treatment and control groups differ significantly in size from the expected ratio, there is a randomization bug that invalidates results. Verify there were no other experiments running on the same traffic that could confound results. Second, statistical validity: ask whether the primary metric was pre-registered or chosen post-hoc (data dredging). Check if the team peeked at results before the pre-determined end date—repeated peeking inflates false positive rates dramatically. If they tested multiple metrics, ask about correction for multiple comparisons using Bonferroni or FDR. Third, effect quality: examine the confidence interval—a statistically significant 5% lift with a confidence interval of [0.1%, 9.9%] is very different from [3.5%, 6.5%]. Check for heterogeneous treatment effects across user segments. Fourth, practical assessment: calculate the expected business impact in absolute terms, check if the effect persists over time (novelty effect), and verify that guardrail metrics like crash rate and latency were not degraded.
Frame your questions as collaborative investigation, not accusation—say "let's verify" rather than "you did it wrong."
Memorize the key pitfalls: peeking, multiple testing, sample ratio mismatch, novelty effects, and underpowered tests.
Demonstrate that you understand both statistical and business significance—a tiny but statistically significant lift may not be worth shipping.
Peeking means checking experiment results before the pre-determined sample size is reached. Each peek inflates the overall false positive rate—checking daily for two weeks at alpha=0.05 can push the actual false positive rate above 25%. Solutions include sequential testing methods like always-valid p-values or group sequential designs that account for multiple looks.
When the metric distribution is heavily skewed (like revenue per user), has heavy tails, or violates normality assumptions. Common alternatives include the Mann-Whitney U test, bootstrap confidence intervals, or transforming the metric (e.g., log transformation). At ByteDance's scale, you can also use quantile-based analysis to understand effects across the distribution.
HireMe AI listens to your interview in real-time, detects questions, and generates instant answer suggestions — so you never freeze up again.
Try HireMe AI Free