Hypothesis driven exploration

In the last post, we looked at collection and analysis of data for pattern discovery using a ride hailing example. An exploratory pass over trip data suggested that demand spikes when it rains, a pattern that looked clean enough to act on immediately. This post is about resisting that urge, and about what it takes to trust a pattern you found while exploring rather than one you set out to test.

The risk with unguided exploration is not that you will fail to find patterns. Given enough columns, enough time windows and enough ways to slice a dataset, you will always find something that looks significant. The risk is that most of what you find this way is noise dressed up as signal. This post covers why that happens, how anchoring exploration to a hypothesis formed before looking at the data keeps analysis honest, and how to tell a real pattern from a coincidence when you were not looking for anything specific to begin with.

The trouble with looking for patterns

Exploratory data analysis is often taught as an unqualified good: plot everything, cross tabulate everything, let the data speak for itself. In practice, an analyst who explores a large dataset without any constraint is running an enormous number of implicit statistical tests, one for every relationship they glance at, and few people account for that scale when judging whether what they see is real. The trip data behind a ride hailing product alone offers weather, day of week, time of day, neighborhood, driver tenure, vehicle type, fare tier, app version and dozens of interactions between them. Look at enough combinations of these and something will cross whatever threshold you use to call a pattern interesting, whether or not anything is actually going on.

The multiple comparisons problem

This is the multiple comparisons problem, and it has a clean statement. If you run $n$ independent comparisons, each at significance level $\alpha$, the probability that at least one turns up significant purely by chance is

$$ P(\text{at least one false positive}) = 1 - (1 - \alpha)^n $$

With $\alpha = 0.05$ and only $n = 20$ comparisons, this probability is already around 64%. An analyst who cuts trip data by neighborhood, weather condition, time of day and driver tenure, then looks at every combination for anything that stands out, is easily running that many comparisons in an afternoon without writing a single line about correcting for it. The comparisons are rarely independent in practice, which changes the exact number, but the direction of the effect does not change: more looks means more chances for noise to masquerade as a finding, and the analyst who does not track how many looks they took has no way of knowing how much to discount what they found.

p hacking and the garden of forking paths

p hacking is the practice of trying multiple analytic choices, which subgroup, which time window, which cutoff, which transformation, until one of them produces a result that clears a significance threshold, then reporting only that result as though it were the only test run.

The garden of forking paths, a term introduced by Gelman and Loken, describes a subtler version of the same problem: even without any intent to cheat, a researcher facing many reasonable analytic decisions, decisions that only become concrete once the data are in hand, can reach a significant result on almost any dataset, because each decision made in good faith is itself an implicit test of the data.

The garden of forking paths version is the one that should worry a data scientist more, because it does not require dishonesty to produce a misleading result. Suppose the rain and demand pattern from the previous post was actually found by looking at weekday trips first, then weekend trips once weekday data looked ambiguous, then narrowing to the neighborhoods with the most trips because the signal was clearer there, then settling on a 30 minute window after trying 15 minutes and an hour. Every one of those choices is defensible on its own, and an analyst who made them sequentially, adapting to what the data showed at each step, would sincerely believe they found one pattern. They actually tested many patterns and kept the one that looked best, which is functionally identical to p hacking even though no single step looks like misconduct.

Anchoring exploration to a hypothesis

The fix is not to stop exploring. Exploration is how the rain and demand pattern was noticed in the first place, and closing off exploration entirely would mean never finding anything you did not already know to look for. The fix is to anchor exploration to a specific hypothesis, even a loosely held one, before digging into the data, and to treat what you find during exploration as a candidate to be confirmed rather than a conclusion to be reported.

Anchoring works because it collapses the effective number of comparisons in the formula above from every combination available in the dataset down to roughly one. A hypothesis stated as trip demand increases after rainfall onset commits you, before you look, to one outcome variable, one direction of effect and roughly one definition of the event under study. You are still free to explore the data to see whether that specific claim holds, but you are no longer free to quietly substitute a different claim halfway through because the original one did not pan out, at least not without being explicit that you have started a new, independent test.

Lightweight pre registration

Academic pre registration is a formal process: before collecting any data, a researcher registers, often on a public repository, the exact hypothesis, the sample, and the analysis plan they intend to run, so that any deviation from the plan is visible after the fact. Few data science teams need that level of ceremony, and most business settings do not have the luxury of collecting fresh data before looking, since the dataset already exists. What is worth borrowing is the underlying discipline in a lighter form.

Before opening the dataset, write down, in a few sentences, what pattern you expect to find, which direction you expect it to run, and what result would count as confirming versus disconfirming that expectation. This does not need to live anywhere formal, a note at the top of a notebook or a message in a project channel is enough. What matters is that the statement exists before you have seen the numbers, because that is what prevents two very human failure modes: quietly reinterpreting a null result as still meaningful once you see it, and treating a marginal or partial result as confirmation because the criteria for confirmation were never pinned down until after the fact. A hypothesis does not need to be held with confidence for this to work, it only needs to be specific. Loosely held and specific are compatible, vague and specific are not.

Separating signal from noise after the fact

Sometimes the useful pattern really was noticed unintentionally, during a general exploratory pass rather than a targeted search, which is exactly how the rain and demand observation surfaced in the previous post. That does not mean it should be discarded. It means the finding needs to be treated differently from a hypothesis that was specified in advance: it needs independent confirmation before it is trusted with the same weight as a planned test, because you cannot rule out that it is one lucky slice among many that were looked at, even if you were not deliberately hunting for one.

Holdout data

The simplest tool is a holdout. Before starting an exploratory pass, split the available historical data into an exploration set and a holdout set that is not touched during exploration. Any pattern noticed while looking at the exploration set is then checked against the holdout set using a single, pre specified test, and only counts as supported if it reappears there. The holdout set functions as the fresh data a formal experiment would collect, without requiring you to wait for new data to arrive.

Replication on a fresh time window

A plain random split of rows works well when observations are independent, but trip demand data is not: trips cluster by day, by weather event and by neighborhood, so a random holdout can still leak the same rain event across both the exploration set and the holdout set. For time ordered, spatially clustered data like this, the safer holdout is a contiguous block of time that was never part of the exploratory pass, for example the four weeks following the window that was originally explored. If the pattern was driven by one unusual storm rather than a general relationship between rain and demand, a fresh window that includes different storms on different days will not reproduce it.

Requiring confirmation out of sample

The general principle behind both of the above is the same: treat an exploratory finding as a hypothesis, not a result, and require it to survive a confirmatory test run on data that played no role in generating it. If the pattern does not reappear, the original finding should be treated as noise, no matter how convincing the original plot looked. A pattern that only exists in the data used to discover it is, by definition, indistinguishable from a pattern that exists only by chance.

Worked example: testing the rain hypothesis

From observation to hypothesis

Picking the thread back up from the previous post: the exploratory pass over ride hailing trip data suggested that demand spikes around rain. That observation, on its own, is not a hypothesis. It is vague about which neighborhoods, vague about timing, and vague about whether the effect is about rain in general or about the moment rain begins. Before touching any more data, that observation gets sharpened into a specific, falsifiable claim.

$$ H: \text{trip demand increases within 30 minutes of rainfall onset in a given neighborhood} $$

Notice what this hypothesis commits to that the original observation did not. It is scoped to a single neighborhood at a time rather than the whole city pooled together, it is scoped to a fixed 30 minute window rather than however long the rain lasts, and it is about the moment rain starts rather than rain in general, since the mechanism under suspicion, riders avoiding walking or losing an alternative mode of transport, should act quickly once rain begins and should not require rain to persist all day. A vague version of this hypothesis would leave room to redefine the window or the geography after seeing the data, which is exactly the garden of forking paths problem from earlier in this post.

Specifying the test before looking

With the hypothesis fixed, the next step is to write down the test before returning to any data that was not already used in the exploratory pass.

Unit of analysis: a single neighborhood observed around a single rainfall onset event.
Treatment definition: a rainfall onset event is a transition from no measurable precipitation to measurable precipitation at the weather station covering that neighborhood.
Outcome: trip requests originating in that neighborhood in the 30 minutes following onset.
Baseline: trip requests in the same neighborhood, same day of week and same time of day, on a comparable set of days with no rain, rather than the 30 minutes immediately before onset, since demand can already be trending for reasons unrelated to weather.
Statistic: the paired difference $\Delta_i = Y_{i,\text{post}} - Y_{i,\text{baseline}}$ for each onset event $i$, averaged across events within a neighborhood.
Decision rule: the hypothesis is considered supported if the average $\Delta$ is positive and a single, paired test, a Wilcoxon signed rank test given that trip counts are unlikely to be normally distributed, returns $p < 0.05$ against the null of no difference. No alternative statistic, window or test is substituted after the result is seen.

Writing this down before looking at new data is the lightweight pre registration described earlier. It is short, it lives in a project note, and it commits the analysis to one test rather than leaving room to pick the best looking one afterward.

Testing against a fresh slice of data

The exploratory pass in the previous post used trip data up to some cutoff date. Rather than accepting the pattern noticed there, the test above is run on a fresh slice: rainfall onset events and matched trip data from the weeks after that cutoff, a period that played no role in forming the hypothesis. This is the same discipline as a holdout set, applied to a time ordered problem by choosing a later window instead of a random sample of rows, for the reasons covered above.

Two outcomes are worth walking through. If the paired difference on the fresh slice is positive and the signed rank test clears the threshold set in advance, the pattern has now been observed in two independent windows using one pre specified procedure, which is much stronger evidence than the original single observation. If the difference is flat or the test does not clear the threshold, the honest conclusion is that the original pattern likely reflected something specific to the exploration window, perhaps a single large storm that coincided with a promotion or a public event that independently raised demand, rather than a general relationship between rain and trip requests.

Interpreting the result

Replication on a fresh window is not proof of causation. Even a pattern that reappears reliably could still be explained by a confound, for example drivers pulling off the road in heavy rain and reducing supply, which would show up as a shift in wait times and prices rather than in the number of requests a rider actually makes. Confirming the pattern out of sample answers a narrower but still essential question: is this a real, stable regularity in the data, or an artifact of the one window we happened to look at first. Only a pattern that clears that bar is worth carrying into the next stage of work, building a model that uses it for prediction, which is the subject of the next post in this series.

Looking ahead

Hypothesis driven exploration is what turns an interesting plot into a claim worth acting on. Once a pattern like the rain and demand relationship survives a test specified before the data was seen, and survives again on a fresh slice it was never allowed to influence, it is ready for the next step in the series: modeling of patterns for prediction, where the question shifts from whether a pattern is real to how much of it a model can actually capture.

References

[1] Gelman, A. and Loken, E. 2013. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p hacking" and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University.

[2] Ioannidis, J.P.A. 2005. Why most published research findings are false. PLoS Medicine. 2, 8 (2005), e124.

Back to posts