Process of Data science - Establishing a baseline

Sep 24, 2021

13 minutes read

Establishing a baseline

In a previous post, generalizing or scaling an experiment from a single pilot to many stores, regions, or customers is discussed. Once a solution is expected to run broadly rather than in one contained setting, the question a data scientist faces changes shape. In the pilot, the question was whether an effect exists at all. At scale, the question becomes whether the model built to exploit that effect is actually worth what it costs to build, run, and maintain, compared to something simpler.

Answering that second question requires a baseline: a plain, easy to compute reference prediction that any sophisticated model has to beat by a meaningful margin before its added complexity is justified. This post covers why a baseline should exist before or alongside a sophisticated model rather than after it, a short list of naive baselines worth trying first, the two ways a baseline can mislead a team (by being too weak or by secretly being too strong), and how the baseline connects to the bias and variance framing from the modeling posts earlier in this series. A worked example on demand forecasting for a retail chain, deciding how much inventory to stock per store per week, ties these ideas together.

Why a baseline comes before the model

A performance number on its own carries almost no information. Ninety percent accuracy, an average error of two units, a lift of five percent, none of these mean anything until they are placed next to a reference point that answers a simple question: compared to what? A baseline is that reference point.

Earlier in this series, the idea of a null hypothesis $H_0$ was introduced as the antithesis a hypothesis $H$ has to overcome before it can be accepted. A baseline plays a closely related role for a predictive model. It is the null model: a plain, transparent, cheap to build predictor that stands in for “nothing clever is happening here.” A sophisticated model is, in effect, a new hypothesis about how well we can predict an outcome, and the baseline is the antithesis it has to beat. Just as a good experimental design settles on the null hypothesis before data is collected, a good baseline should be settled on before, or at the very latest alongside, the sophisticated model, not constructed afterward to make a finished project look good.

There are a few reasons ordering matters this way. First, choosing a baseline after seeing the sophisticated model’s results invites the team to pick, consciously or not, whichever reference point makes the new work look most impressive. Second, a baseline forces the team to state, in plain terms, what a reasonable person already believes about the outcome before any modeling starts, which is valuable context for stakeholders who are being asked to trust a more opaque system. Third, a baseline is often good enough to ship, or close enough that it buys time to build the sophisticated model properly rather than under pressure.

Naive baselines worth trying first

A useful baseline is not the best possible simple model, it is the simplest model a reasonable person would try first. A few patterns cover most situations a data scientist runs into.

The historical average

Predict the mean (or median, for skewed outcomes) of the outcome observed historically, ignoring any input features entirely.

$$ \hat{y}^{avg} = \frac{1}{T}\sum_{i=1}^{T} y_i $$

This baseline answers the question: how much better than “always guess the average” is the sophisticated model? It is the right first check whenever a stakeholder’s intuition about the new model’s value is anchored on how far its predictions are from the truth in absolute terms, without a sense of how variable the outcome already is.

No change from last period

Predict that the next value equals the most recently observed value. This is often called a persistence baseline in forecasting contexts.

$$ \hat{y}_t^{persist} = y_{t-1} $$

Persistence is a strong baseline whenever an outcome changes slowly relative to how often it is measured, for example short term demand, sensor readings, or prices. It is a weak baseline whenever the outcome has strong periodic structure, a point returned to in the worked example below, where a seasonal variant of this idea does a much better job.

A rule of thumb a human already uses

Before a data scientist arrives, someone was already making this decision, usually with a simple heuristic learned through years of trial and error. A claims adjuster might flag anything above a round dollar threshold, a planner might order enough stock to cover last period’s sales plus a fixed safety margin, a nurse might triage by a short checklist. That heuristic is a baseline too, and often the most important one, because it is the thing the sophisticated model is actually going to replace in practice. If the model cannot beat the heuristic a person is already applying, it will be a hard sell regardless of what it does against a historical average.

A useful habit is to interview the practitioner directly and write their rule down as an explicit, runnable procedure before writing a single line of modeling code. This mirrors the point made in the post on measurement variables about targeted interviews providing priors for data gathering. Here the interview provides a prior for what counts as success.

When a baseline is too weak

A baseline that ignores information a practitioner already has access to sets the bar too low. If demand is highly seasonal and the chosen baseline predicts a single global average with no adjustment for the season, almost any model that includes a calendar feature will look like a breakthrough, when in fact it is only catching up to what a seasonally aware human already knew. The same failure mode shows up in classification problems with an imbalanced outcome: a model that predicts the majority class every time can post a deceptively high accuracy figure, and a sophisticated model that edges past it by a point or two of accuracy can look like a large win while still being nearly useless at the thing that actually matters, correctly flagging the minority outcome.

The fix is to always pick the strongest naive baseline that is fair to use, not the weakest one that is easy to beat. If a seasonal pattern is known, the baseline should include it. If a practitioner’s rule of thumb is available, it should be measured, not assumed. A model that only clears a weak baseline has not yet earned the cost of its complexity, it has only exposed that the weak baseline was a poor choice of comparison.

When a baseline is too strong: a leakage problem

The opposite failure is quieter and more dangerous because it looks like rigor rather than sloppiness. A baseline can be constructed, often with good intentions, using information that would not actually be available at the moment a real prediction is needed. This is a form of leakage, the same underlying problem discussed in the post on numerical analysis and error reduction, except here it contaminates the reference point rather than the candidate model, which makes it easy to miss because teams tend to scrutinize the sophisticated model’s features far more carefully than the baseline’s.

A few concrete ways this happens. A centered moving average smooths each period using values from both before and after it, which requires knowing the future relative to the point being predicted. A baseline built from “this period’s eventual total, divided evenly” bakes in information that only exists once the period is already over. A baseline tuned or selected using the same holdout data that will later be used to judge the sophisticated model gives the baseline an unfair look at the answer key.

When a baseline like this is used, it can make even a carefully built model look weak or unnecessary, because the comparison is not fair to begin with. The sophisticated model is being asked to beat a reference point that could never actually be run in production the way it was run in the evaluation. The practical guard against this is simple to state and easy to skip under time pressure: every baseline, no matter how naive, should be checked against the same rule applied to the sophisticated model, namely that it only uses information that would genuinely have been on hand at prediction time.

The baseline as a lower bound

Put together, a fair baseline gives a lower bound: the sophisticated model is worth adopting only if it clears that bound by more than could plausibly be explained by noise, and by enough to justify what the added complexity costs to build and keep running.

This connects directly to the bias and variance framing introduced in the post on modeling patterns. A naive baseline typically sits at one extreme: high bias, because it ignores structure a more flexible model could capture, but low variance, because it has few or no parameters to overfit and behaves predictably from one sample of data to the next. A sophisticated model moves along that tradeoff, reducing bias by capturing nonlinearities, interactions, or finer grained structure, at the cost of variance, since a more flexible model is more sensitive to the particular sample it was trained on and more likely to fit noise along with signal.

$$ \text{Error} \approx \text{Bias}^2 + \text{Variance} + \text{Irreducible error} $$

A baseline lets a data scientist see whether the bias reduction from added flexibility is actually large enough to be worth the variance and operational cost that comes with it, rather than trusting a lower training error at face value. A simple decision rule follows from this: adopt the sophisticated model only when

$$ \text{Error}_{baseline} - \text{Error}_{model} > \Delta $$

where $\Delta$ is not zero, but a margin large enough to cover sampling noise in the comparison plus the extra cost of building, explaining, and maintaining the more complex system. Choosing $\Delta$ honestly, rather than accepting any improvement no matter how small, is most of what separates a useful baseline exercise from a box ticking one.

Worked example: demand forecasting for a retail chain

The forecasting problem

Consider a retail chain that needs, every week, a forecast of demand for each product category at each store, in order to decide how much inventory to ship and stock for the coming week. Order too little and the store runs out, losing sales and, over time, customer goodwill. Order too much and the excess sits in a stockroom tying up capital, or for perishable categories, is written off entirely. The forecast feeds a decision with real cost on both sides of the error, which makes it a natural place to insist on a baseline before signing off on a sophisticated forecasting system.

The naive seasonal baseline

A plain persistence baseline, this week equals last week, is a poor fit here because retail demand carries strong yearly seasonality: holidays, back to school periods, and weather driven categories all swing demand well beyond what last week’s number would suggest. A fairer naive baseline accounts for that seasonality directly by predicting that this week’s demand equals the same week last year.

$$ \hat{y}_{s,t} = y_{s,t-52} $$

for store and category $s$ and week $t$, using a fifty two week cycle. This seasonal naive baseline costs almost nothing to compute, needs no feature pipeline, and already captures the single largest source of predictable variation in retail demand. It is a much harder target to beat than plain persistence, and beating it is a meaningful achievement precisely because it is not a weak baseline.

The sophisticated model

A more sophisticated approach might use gradient boosted trees or a hierarchical forecasting model trained across all stores and categories at once, pooling information across similar stores, with features such as the promotions calendar, current price and any planned price changes, a weather forecast for the target week, local events, and recent foot traffic trends. Pooling data across stores in particular gives this model a chance to reduce bias in ways the per store seasonal baseline cannot, by borrowing patterns learned from similar stores to sharpen forecasts for stores with thinner history, such as recently opened locations.

Comparing the two honestly

A fair comparison requires a backtesting protocol built around a rolling origin: for each historical week used as an evaluation point, both the seasonal baseline and the sophisticated model are given only the data that would have existed before that week, and are judged on what actually happened afterward. The promotions calendar and weather must be the version that was known or forecast ahead of time, not the finalized version recorded after the fact, otherwise the sophisticated model is being handed the same kind of leakage discussed earlier, except now on the model side instead of the baseline side, which would make the comparison look better than it would ever perform in production.

Suppose the resulting comparison, measured as weighted absolute percentage error averaged across stores and categories, shows the sophisticated model beating the seasonal baseline by fifteen percent on average. That sounds like a clear win, but two further checks matter before rolling it out everywhere. First, is the fifteen percent gain spread evenly, or is it concentrated in a subset of high volume stores and categories where enough history and cross store signal exists for the model to learn something real, while thin history locations show little gain or, in a few cases where a new store recently opened or a supply disruption hit an unusual pattern, the sophisticated model does noticeably worse than the seasonal baseline because it extrapolates confidently from too little relevant history. The seasonal baseline, having no such flexibility, simply falls back to what a similar week looked like before, which tends to be a safer failure mode than a confident but wrong extrapolation.

Is the complexity earning its keep

The final step is translating the accuracy gain into the terms that actually matter: the cost of a stockout against the cost of holding or writing off excess inventory, converted into a dollar figure per store per week. If the fifteen percent reduction in forecast error translates into a meaningful reduction in stockouts and markdowns, and that saving clearly exceeds what it costs to build, monitor, and retrain the sophisticated pipeline (data engineering for the extra features, a retraining schedule, monitoring for drift, and a process for store managers to override the system when it clearly misses something local knowledge would have caught), then the added complexity has earned its keep and a broad rollout is justified.

If instead the gain is concentrated in a handful of high volume stores, a more targeted decision follows naturally from the same comparison: run the sophisticated model where the pooled data gives it a genuine edge, and let the cheap, robust seasonal baseline continue to handle the long tail of smaller stores and thinner categories where it is already close to as good and far less likely to fail in an unusual way. This segmented outcome, choosing where complexity pays for itself rather than treating the decision as all or nothing, is usually the most defensible answer a baseline comparison produces in practice.

Whichever way the rollout goes, the comparison is not a one time exercise. Demand patterns drift, new stores open, and a model or baseline that cleared the bar convincingly at launch can quietly stop doing so months later. That ongoing comparison against a baseline, maintained as a permanent part of production monitoring rather than a one off study, is the subject of the next post in this series.

References

[1] Hyndman, R.J. and Athanasopoulos, G. 2018. Forecasting: Principles and Practice. 2nd edition. OTexts.

Back to posts