Kiran Kumar
Experimental design (0 to 1)
In the last post we looked at latent or unobservable factors, the hidden variables that can quietly shape an outcome even when they never show up directly in a dataset. Once a hypothesis is framed and the measurement variables and latent factors around it are understood, the next question a data scientist has to answer is a practical one: how do we actually test this thing? This post is about the first experiment, the smallest and cheapest test that can tell us whether an idea is worth pursuing further before any scaling, tooling, or automation gets built around it.
I am reusing the "zero to one" framing from the same source cited in the Hypothesis post, Peter Thiel's Zero to One. The book's core argument is that going from nothing to something (zero to one) is a fundamentally different problem than taking something that already works and producing more of it (one to n). Experimental design at this stage in the process is squarely a zero to one problem. The goal is not to stand up a full experimentation platform with dashboards and automated rollouts. The goal is to answer one question honestly, with the least amount of effort that still produces a trustworthy answer.
- Hypothesis
- Measurement variables
- Latent or unobservable factors
- Experimental design (0 to 1) (discussed here)
- Collection and analysis of data for pattern discovery
- Modeling of patterns for prediction
- Generalizing or scaling the experiment (1 to n)
- Establishing a baseline
- Monitoring through controls and baselines
- Ethics and governance
What makes a good minimal experiment
A minimal experiment earns that name by being narrow in scope, not careless in method. Keeping the design small is what makes it fast and cheap to run, but the design still has to be sound enough that the result actually means something. Three ingredients separate a useful minimal experiment from a wasted one: a single clear question, a single primary metric, and a defined population.
A single clear question
A good first experiment tests one hypothesis, not a bundle of related ideas. "Does a simplified checkout increase completed purchases" is a single question. "Does a simplified checkout, a new homepage banner, and a loyalty prompt together improve the business" is three questions wearing a trench coat. Each of those needs its own test, run and read on its own terms, or the result will not tell you which change did anything. If the temptation to combine ideas is strong, that is usually a sign the underlying hypothesis has not been narrowed enough yet, which is exactly the work item 1 and item 3 in this series are meant to resolve before you get here.
A single primary metric
Before the experiment starts, pick the one number that will decide whether the hypothesis is supported. Everything else is a secondary or guardrail metric, useful for context and for catching unintended harm, but not a metric you get to promote to "the reason it worked" after the fact. Deciding this ahead of time protects against a common failure mode where a team runs a test, the primary metric does not move, and someone finds a different metric among a dozen that moved by chance and declares success. With enough metrics on a dashboard, some of them will look significant purely due to random noise, so the discipline has to be front loaded, not applied retroactively.
A defined population
Who is actually eligible for this experiment, and over what window of time? A defined population states clearly who is included, who is excluded, and why. It should match, as closely as possible, the population the finding will eventually be applied to. Testing a change only on your most active, most loyal customers and then rolling it out to everyone is a common way to overestimate an effect, because the population that opted into the test does not resemble the population that will experience the change later.
Randomization and why it matters
Randomization is the mechanism that makes a clean comparison possible. Assign each unit in the population to a variant using a process with no relationship to who that unit is or how they might behave, and in expectation the two groups end up statistically identical on every factor that could matter, the ones you measured and the ones you did not think to measure. This is the practical answer to the latent factor problem from the last post: you do not have to enumerate every hidden confound if random assignment has already balanced it across groups for you.
Compare this to letting people opt in to a new experience, say an early access toggle for the new checkout. The group that opts in is different from the group that does not before the experience even begins: they are probably more engaged, more tolerant of change, or more tech comfortable. Any difference you observe afterward is now tangled up with those preexisting differences, and there is no clean way to pull the two apart after the fact. This is the classic selection bias problem, and no amount of clever modeling after the data is collected fully undoes it.
In practice, randomization is usually implemented deterministically: a hash of a user id or session id is mapped into a fixed range, and a threshold on that range decides the assignment. The same input always produces the same assignment, which is what lets a returning visitor see a consistent experience without the system needing to store a separate assignment record for every unit up front.
Minimum viable experiment versus a fully powered study
A minimum viable experiment is the smallest test that could plausibly support or undercut the hypothesis. It runs on a modest sample, over a short window, aimed at one question, and it is allowed to sacrifice some precision in exchange for speed. Its job is to tell you whether an idea clears the bar of "worth investing further in," not to produce a publication grade estimate of the exact size of the effect.
A fully powered study is a different animal, deliberately sized in advance using a target effect size, a baseline rate, and a chosen confidence and power level, so that it can make a precise, defensible claim about the size of an effect and support a decision to commit real resources at scale. That kind of rigor belongs later in the process, closer to item 7 in this series, when an idea that already looked promising in a minimal experiment is being generalized from a handful of sessions to the full population.
Running a fully powered study before you have any evidence the idea works at all is expensive premature optimization, the same trap called out in the Hypothesis post about jumping straight to scaling before solving the zero to one problem. Running only ever small, underpowered tests forever, and never following up with a properly sized study before a big rollout, is the opposite mistake: you end up making expensive decisions on noisy evidence.
Statistical power and sample size in plain language
Every metric measured on a sample bounces around a little, purely from who happened to land in that sample, even when there is no real effect underneath it at all. Flip a fair coin ten times and you would not be shocked to see seven heads. Flip it ten thousand times and a result of seventy percent heads would be extraordinary. The same logic applies to a conversion rate, an average order value, or any other metric: small samples are noisy, large samples are calm, and the calming happens because individual quirks average out as more of them pile up.
Statistical power is just the probability that your test will actually detect a real effect of a given size, if that effect truly exists. A small sample is not just imprecise, it is underpowered: a true, modest sized improvement can easily get lost inside the normal noise of the metric, and the experiment reports "no significant difference" even though the treatment genuinely helped. Sample size and effect size trade against each other. A large, obvious effect can be detected with a small sample because it stands out well above the noise. A small, subtle effect needs a much bigger sample before it can be reliably told apart from ordinary variation.
You do not need the full derivation to use this intuition, but a rough rule of thumb helps make it concrete. For comparing two proportions (say, a completion rate under a control and a treatment), a commonly cited back of the envelope approximation for the sample size needed per group, at conventional significance and power levels, is
$$ n \approx \frac{16 \cdot p(1-p)}{\delta^2} $$
where $p$ is the baseline rate and $\delta$ is the absolute difference you want to be able to detect. Treat this as a sanity check, not a substitute for a proper power calculation, but notice what it implies: $\delta$ is squared in the denominator, so cutting the effect size you want to detect in half roughly quadruples the sample you need. This is the mathematical reason small early experiments should aim to detect large, obvious effects, and should not be expected to reliably confirm tiny ones.
Common early mistakes
Testing too many things at once
Bundling several changes into one variant, or running many variants against a single control, spreads a fixed amount of traffic thin and multiplies the number of comparisons being made. Both effects work against you: less traffic per arm weakens power, and more comparisons raises the odds that something looks significant purely by chance. If several ideas seem worth testing, that is a sequencing problem to solve before launch, not something to solve by cramming everything into one test.
No clear success metric
Without a primary metric fixed in advance, it is tempting to look at a dashboard full of numbers after the test ends and build a story around whichever one moved. This pattern, sometimes called hypothesizing after the results are known, produces conclusions that feel justified but were never actually tested with any statistical guarantee. The metric has to be chosen before the data is seen, not after.
No plan for how long to run it
Checking results every day and stopping the moment the numbers look good is one of the most common ways an early experiment misleads a team. Random noise crosses a significance threshold far more often than people expect if you give it enough chances to do so, and checking daily gives it many chances. Decide the run length before starting, based on the sample size you need and how long it takes to reach it, and commit to reading the result only once that window has passed.
Worked example: a simplified checkout flow at an online store
Suppose an online store's product team believes that its checkout, currently spread across four separate pages (shipping address, shipping method, payment, and review), is losing customers to friction and abandonment. The hypothesis is:
Replacing the four step checkout with a single, simplified one page checkout increases the rate of completed purchases among shoppers who reach checkout.
Picking the primary metric
The natural primary metric is completed purchase rate, defined as purchases divided by checkout starts: the fraction of shoppers who reach the first checkout step and go on to place an order. This is preferred over a metric like total revenue, which is noisier session to session because a handful of large orders can swing it independently of whether the checkout flow itself is working. Average order value and page load time are tracked as guardrail metrics, watched to catch unintended harm, but neither is allowed to substitute for the primary metric when it comes time to call the test a success or a failure.
Defining the population
The population is every shopper who reaches the first checkout step during the test window, across both web and mobile, excluding known bot and internal employee traffic. Shoppers who never reach checkout are outside the population entirely, since the change under test only touches what happens after that point, and including them would just dilute the metric with sessions the change could never have affected.
Choosing the randomization unit: session versus user
A real design choice sits here: randomize by session or by user? Session level randomization is simple and works even for shoppers who are not logged in, and it assigns a variant right at the moment a shopper reaches checkout. Its weakness is that a returning shopper could see the old checkout on one visit and the new one on another, and if a shopper's memory of the previous experience carries over (for example, familiarity with a flow they have used before, or frustration from an earlier visit), comparisons across sessions get contaminated by that carryover.
User level randomization keeps a single shopper on the same variant for as long as the test runs, using a logged in account id where available and a persistent cookie otherwise. It costs a bit more engineering, and it does not work as cleanly for genuinely anonymous, cookie clearing traffic, but it avoids the carryover problem and its unit of analysis matches its unit of randomization, which keeps the statistics honest. For a checkout change, where prior familiarity with the flow plausibly affects the outcome, user level randomization is the better choice, with session level treated as a fallback for shoppers with no persistent identifier available.
How long to run the test
Say the store's historical completion rate from checkout start to purchase is twenty percent, and the team wants to be able to detect an absolute improvement of two percentage points, from twenty percent to twenty two percent. Plugging $p = 0.20$ and $\delta = 0.02$ into the rough sample size guide from earlier:
$$ n \approx \frac{16 \times 0.20 \times 0.80}{0.02^2} = \frac{2.56}{0.0004} \approx 6400 $$
That is roughly six thousand four hundred checkout starts needed in each of the two groups, about twelve thousand eight hundred in total. If the store sees around two thousand checkout starts a day, the sample requirement alone would be met in under a week, but the run length should not stop there. Purchasing behavior varies by day of week (weekday shoppers are not the same population as weekend shoppers), so the test should run for at least one full week, and ideally two, so both weekday and weekend traffic are represented in a stable ratio. The team commits to that duration before launch, resists the urge to peek and stop early even if early numbers look promising, and only reads the primary metric once the planned window has closed.
A test like this, sized and scoped this way, is deliberately narrow. It answers exactly one question about exactly one metric for a well defined population. That narrowness is a feature at this stage, not a limitation: the whole point of a zero to one experiment is to get a fast, honest read on whether the idea deserves further investment, before any of the fully powered rollout machinery from later in this series gets built around it.
The next post in this series looks at the other side of this same experiment: controlling for the other factors running alongside a test like this one, so that the effect you observe can be attributed to the change you actually made.