什么是Sample size？

Sample size is the number of visitors (and, more importantly, conversions) an experiment needs before its result means anything. Small samples are wildly noisy: at 50 visitors per variant, a few chance conversions produce huge apparent differences, which is why an A/B test that looks decisive on day two so often evaporates by day fourteen.

The required size is calculable in advance from three inputs: your baseline conversion rate, the smallest lift you care to detect, and your desired confidence and power (conventionally 95 percent and 80 percent). The arithmetic is sobering for small sites. Detecting a relative 10 percent lift on a 3 percent baseline needs on the order of 50,000 visitors per variant; detecting a 50 percent lift needs a few thousand. Free calculators do the math; the discipline is doing it before launch and committing to the number, since stopping early because the result "looks done" is exactly the peeking problem that invalidates significance.

If the required sample would take six months to collect, the answer is not a longer test but a different one: test bigger changes, test a higher-traffic page, or measure a more frequent micro conversion.