Every PM eventually gets asked: what did the data show? A/B testing is how you replace "I think this will work" with "the data shows this works." It is one of the most powerful tools in a PM toolkit — and one of the most commonly misused. Here is how to design, run, and interpret experiments without needing a statistics degree.
The 4-question checklist before running a test
Before you run any experiment, answer four questions. First: do we have enough traffic? A test that runs for six months to reach statistical significance is usually not worth running. Second: do we have a clear hypothesis? "Let us try a different button color" is not a hypothesis. Third: can we actually measure the outcome we care about? If the metric you want to move is not tracked, fix the tracking first. Fourth: can we commit to running this test to completion? Stopping a test early because it looks bad is one of the most common ways experiments go wrong.
How to write a hypothesis
A strong A/B test hypothesis has three parts: the change you are making, the outcome you expect, and the reason you believe the change will produce that outcome. For example: "If we move the call-to-action button above the fold, we expect checkout conversion to increase because users who do not scroll will now see the primary action." That structure forces you to think through causality rather than just picking changes to test at random.
Primary metrics vs. guardrail metrics
A well-designed experiment tests one thing. Your primary metric is the outcome you are trying to move. Your guardrail metrics are the things you want to make sure you are not breaking while you move the primary metric. A checkout experiment might have conversion rate as the primary metric and average order value, return rate, and page load time as guardrails. If your primary metric improves but a guardrail degrades significantly, you do not have a win — you have a tradeoff that needs more scrutiny.
What statistical significance actually means
A result is statistically significant at p less than 0.05 when there is less than a 5 percent chance that the difference you observed happened by random chance. In plain English: you are 95 percent confident the result is real. That is the standard threshold most teams use. It does not mean the result is large enough to matter — a statistically significant 0.1 percent improvement in conversion may not justify the engineering cost of shipping the change. Significance and magnitude are two separate questions.
The mistakes that invalidate experiments
Peeking — checking your results before the test is complete and stopping it early when you like what you see — inflates your false positive rate significantly. Running a test for less than one full business cycle (usually at least one week, ideally two) means your results may reflect day-of-week patterns rather than real behavior. Running multiple changes in one test makes it impossible to know which change drove the result. These mistakes do not just produce bad data — they produce confidently wrong data, which is worse.
How to communicate results to stakeholders
Structure your results readout in three parts. The headline: did it work? A clear yes, no, or inconclusive. The context: by how much did the primary metric move, and with what confidence level? What happened to the guardrail metrics? The recommendation: ship it, iterate and retest, or revert? Stakeholders do not want a statistics lecture — they want a clear answer and a clear next step. Give them that, and have the details ready if they ask.