In brief: A/B testing compares versions exposed to comparable audiences to estimate the effect of one change.
How I use A/B testing in practice
I define one hypothesis, primary metric, audience, sample and decision rule before launch. I keep other important conditions stable and record the test period.
What to watch
Stopping early, changing several things or testing too little traffic creates false confidence. Statistical significance does not guarantee business significance.
Questions for decisions
- What single change is being tested?
- Is the sample sufficient?
- What result would justify rollout?
- Could the effect be seasonal or accidental?