Kiran Kumar
Monitoring through controls and baselines
In the last post we discussed establishing a baseline once a model has been generalized from a single experiment to a full scale rollout. A baseline is a snapshot: the error rate, the distribution of inputs, and the business metric a model is expected to hold on the day it goes live. That baseline was framed as the answer to the question "how good is this model right now, and against what do we compare it later." This post is about the "later" part. A model does not stay still after it ships. The world around it keeps moving, and the baseline we worked so hard to establish is also the yardstick we monitor against for as long as the model stays in production.
The example carried forward from the previous post is a retail demand forecasting model, one that predicts how many units of each product a store should expect to sell in the coming week so that ordering and inventory decisions can be made ahead of time. The baseline post established what "good" looked like at launch. This post walks through what happens a few months later, when shopping patterns in one region shift abruptly and the model keeps producing forecasts as if nothing had changed.
- Hypothesis
- Measurement variables
- Latent or unobservable factors
- Experimental design (0 to 1)
- Collection and analysis of data for pattern discovery
- Modeling of patterns for prediction
- Generalizing or scaling the experiment (1 to n)
- Establishing a baseline
- Monitoring through controls and baselines (discussed here)
- Ethics and governance
Deployment is not the finish line
It is tempting to treat shipping a model as the last step of a project. The hypothesis was formed, variables were measured, confounds were controlled, patterns were modeled, the experiment was scaled from a pilot to full coverage, and a baseline was signed off. Surely the hard part is over. In practice, deployment is closer to the midpoint. A model making decisions in production is now exposed to a stream of new data it was never trained on, and to a world that has every incentive to keep changing regardless of whether the model is ready for it.
Two related but distinct things can go wrong after launch, and separating them clearly matters because the fix for each is different.
Data drift and concept drift
Let $X$ be the inputs a model consumes (in the forecasting example: recent sales, price, promotions, seasonality, local events) and $Y$ be the quantity being predicted (units sold next week). A model learns an approximation of the conditional relationship $P(Y \mid X)$ from historical data, using the observed distribution of inputs $P(X)$ along the way.
Data drift
Data drift, sometimes called covariate shift, occurs when $P(X)$ changes while the underlying relationship $P(Y \mid X)$ stays the same. The model's learned function is still valid, but it is now being asked to make predictions on inputs that look different from what it saw during training.
$$ \text{Data drift: } P_{train}(X) \neq P_{live}(X), \quad P(Y \mid X) \text{ unchanged} $$
In the forecasting example, data drift would show up as a change in the mix of products being sold, a change in average promotion frequency, or a new store opening with a different customer profile than the stores the model was trained on. The relationship between "sales history plus price plus season" and "next week's demand" has not necessarily broken, but the model is now extrapolating into territory it has less experience with, and its error tends to creep up as a result.
Concept drift
Concept drift is more serious. It occurs when the relationship $P(Y \mid X)$ itself changes, even if the inputs look statistically similar to what the model saw during training.
$$ \text{Concept drift: } P(Y \mid X)_{train} \neq P(Y \mid X)_{live} $$
In the forecasting example, concept drift would show up if the same combination of recent sales, price, and season now leads to a different demand outcome than it used to, because the reason people buy has changed, not just how much they buy of what. A regional supply disruption at a competitor, a new store opening nearby, or a change in commuting patterns after a local employer relocates can all rewrite the relationship between the signals the model watches and the demand that follows. No amount of data volume fixes concept drift, because the pattern the model memorized is genuinely out of date. The model needs to relearn the relationship, which usually means retraining on more recent data or adding a feature that captures the new regime directly.
Distinguishing the two matters because a monitoring system that only checks whether inputs look normal (a data drift check) can miss concept drift entirely: the inputs can look perfectly ordinary right up until the moment the world attaches a different outcome to them. This is why the practice described in the rest of this post pairs input monitoring with monitoring the model's actual error, since error is the signal that ultimately captures both kinds of drift.
Control chart style monitoring
The simplest and most durable form of production monitoring borrows an idea from manufacturing quality control that predates modern data science by close to a century: the control chart, introduced by Walter Shewhart at Bell Labs in the 1920s. The idea is to track a metric over time, establish the range it is expected to fall within based on historical behavior, and flag any point that falls outside that range as a signal worth investigating rather than noise to ignore.
Applied to a deployed model, the metric being tracked is usually the error of the model's predictions, measured on a rolling basis (daily or weekly, depending on how quickly the process being modeled changes). For the forecasting example, a natural choice is weighted absolute percentage error $WAPE$ computed each week across all stores and products:
$$ WAPE_t = \frac{\sum_i |y_{i,t} - \hat{y}_{i,t}|}{\sum_i y_{i,t}} $$
where $t$ indexes the week, $i$ indexes a store and product combination, $y_{i,t}$ is actual demand, and $\hat{y}_{i,t}$ is the forecast. The baseline established at launch provides the mean $\mu_0$ and standard deviation $\sigma_0$ of this metric under normal operating conditions, observed over enough weeks to be a stable estimate rather than a lucky or unlucky stretch. Control limits are then set some number of standard deviations away from that mean, commonly three, matching the traditional Shewhart convention:
$$ UCL = \mu_0 + 3\sigma_0, \qquad LCL = \max(0, \mu_0 - 3\sigma_0) $$
Each new week's $WAPE_t$ is plotted against these fixed limits. A single point outside the limits could be noise (a holiday, a data pipeline hiccup, one unusually large order), so most practical setups add a rule that only fires an alert when the metric breaches the limit and stays there, or when several consecutive points trend in the same direction even while still inside the limits (a run of eight increasing points is a classic rule of thumb borrowed from the same manufacturing literature, since a monotonic run that long is unlikely under normal random variation).
Control charts are attractive because they are simple, cheap to compute, and easy to explain to a business stakeholder who is not going to sit through a lecture on statistical drift tests. They are not the only tool available. Heavier weight approaches such as the population stability index or a Kolmogorov Smirnov test compare the full shape of the live input distribution against the training distribution and can catch data drift before it shows up in the error metric at all. These are worth adding for a mature monitoring system, but a control chart on the error metric is the minimum viable version and, in my experience, catches the majority of problems that actually matter to the business, because it monitors the thing decisions are made from.
Two baselines: the original and the rolling one
A subtlety worth calling out is that "compare against the baseline" can mean two different things, and a mature monitoring setup tracks both.
Comparing against the fixed launch baseline
The first comparison is against the fixed baseline established at launch, $\mu_0$ from the previous section, which never moves. This answers the question "are we still delivering what we promised the business when this model went live." It matters because a model can degrade so gradually, week over week, that no single week looks alarming, yet a year later the model is delivering meaningfully worse forecasts than it did on day one. A fixed baseline anchors long term accountability and is the number that belongs in a quarterly business review.
Comparing against the model's own performance at launch time
The second comparison is a rolling one: how is the model doing this month compared to how it was doing three months ago, regardless of where the fixed baseline sits. This is a different question, closer to "has something changed recently," and it is the one that catches an abrupt regime change quickly. A model that has always run a little worse than its original launch target, but has been stable at that level for a year, does not need a fire drill. A model that suddenly starts producing errors twice as large as it did last month does, even if the fixed baseline comparison has not yet crossed a dramatic threshold because the degradation only started recently.
Running both comparisons side by side avoids two opposite failure modes: treating every small deviation from the ambitious original baseline as an emergency (which trains the team to ignore alerts), and missing a real, recent shift because the model's average performance over its whole lifetime still looks acceptable.
What a good alerting and response process looks like
A control chart with nobody watching it is decoration. The value of monitoring comes entirely from what happens in the minutes and days after a control limit is breached, so it is worth being explicit about the process rather than leaving it implicit.
Who gets notified
A breach of the error control limit should route to a named owner, not a shared inbox that nobody checks on a Friday afternoon. For a forecasting model feeding inventory decisions, that is typically the data scientist or machine learning engineer who owns the model plus the operations or supply chain manager who consumes its output, since the operations side often has context (a new competitor, a road closure, a supplier issue) that explains what the numbers alone cannot.
Triage before reaction
The first question after an alert is not "how do we fix the model" but "what kind of drift is this." Is the breach isolated to one region or product category, suggesting a local, explainable cause, or is it broad, suggesting a pipeline bug or a systemic shift. Is it a single bad week that could be a data quality issue upstream, or a sustained trend consistent with genuine data or concept drift. This triage step usually takes an hour or two and prevents a team from retraining a model in a panic over what turns out to be a broken feature pipeline.
The rollback plan
Every deployed model should ship with a rollback plan decided in advance, not improvised under pressure. For a forecasting model this is usually one of a small number of options: fall back to a simpler heuristic (such as a moving average or the same week last year, adjusted for a known trend) for the affected region while the underlying model is investigated, revert to the previous model version if the new behavior started right after a recent redeployment, or apply a manual override supplied by the operations team for the specific stores affected while a fix is developed. The plan should specify who has the authority to trigger the rollback and how quickly it can be executed, because the cost of a bad forecast compounds every day it goes uncorrected, either as a stockout that loses sales or as an overstock that ties up capital and shelf space.
Closing the loop
Once the immediate issue is handled, the event belongs in a short written record: what triggered the alert, what the root cause turned out to be, what action was taken, and whether the control limits or the monitored metric should be adjusted as a result. Over time this record is what turns monitoring from a reactive fire alarm into a source of institutional memory about how the model's environment tends to change.
Worked example: when shopping patterns shift under a demand forecast
Picking the retail demand forecasting model back up from the previous post, recall the baseline established at launch: weekly $WAPE$ across all stores and products averaged around 12 percent, with a standard deviation of about 1.5 percentage points over the first quarter of stable operation. Using the three sigma convention from earlier, this puts the upper control limit at roughly 16.5 percent. Below that line, week to week wobble is expected and ignored. Above it, someone should look.
The trigger
Four months after launch, a large competitor opens a store two blocks from one of the retailer's busiest locations, and around the same time a supplier disruption affects a category of goods carried heavily in that same region. Together these two events change local shopping habits: some customers who used to buy staple goods at this store now split their trips with the new competitor, while a shortage in one product category pushes customers toward substitute products the model has little history forecasting well. Neither event shows up as a labeled feature anywhere in the model's input data. From the model's point of view, recent sales history simply starts looking unusual relative to what it learned, and in the affected product category the relationship between recent sales and next week's demand has genuinely changed, a combination of data drift in the inputs and concept drift in the category most exposed to the supply disruption.
How monitoring catches it
In the first week after the competitor opens, regional $WAPE$ ticks up to 15 percent, still inside the control limit, and is noted but not escalated. The following week it climbs to 17.5 percent, breaching the upper control limit for the affected region, and the week after that it holds at 19 percent. That is two consecutive weeks outside the limit, which is enough under the response rules established earlier to trigger an alert rather than being dismissed as a single noisy week. The rolling comparison against the model's own performance three months prior confirms the alert is meaningful. Company wide $WAPE$ has barely moved, but the affected region's error has risen by more than five percentage points in three weeks, a much sharper move than the slow drift the fixed baseline comparison alone would have flagged months later.
Triage and response
The alert routes to the model owner and the regional operations manager. The regional breakdown in the monitoring dashboard immediately narrows the problem: it is isolated to one region and concentrated in one product category, ruling out a pipeline bug (which would typically show up everywhere at once) and pointing toward a local, explainable cause. A short conversation with the operations manager surfaces both the new competitor store and the supplier disruption within the hour, well before a sophisticated statistical test would have been needed to diagnose the mechanism.
With the cause identified as a genuine regime change rather than a data quality issue, the response follows the plan agreed in advance. As an immediate stopgap, the affected stores are switched to a manual override informed by the operations team's judgment while a fix is developed, avoiding another two or three weeks of forecasts generated from a relationship that no longer holds. In parallel, the model is retrained on the most recent data, which now includes several weeks of the new pattern, and a new feature flagging proximity to a competing store within a set radius is added so that future changes in the competitive landscape are represented explicitly rather than left for the model to infer after the fact. The retrained model is validated against the last several weeks of actuals before being rolled back into production, and the region's control limits are recalibrated once performance stabilizes under the new pattern.
What monitoring prevented
Without this process, the model would have kept generating forecasts based on a relationship that stopped holding the moment the competitor opened. Two outcomes were realistically at stake depending on the direction of the error. Under forecasting the category affected by the supplier disruption, given that customers were substituting toward it, would have led to a stockout, lost sales, and disappointed customers turning elsewhere. Over forecasting demand at the store now splitting traffic with the new competitor would have led to excess inventory sitting on shelves, tying up capital and eventually requiring markdowns. Catching the shift within three weeks, instead of discovering it a quarter later during a routine business review, is the entire value proposition of monitoring: it converts a slow, expensive surprise into a fast, cheap correction.
Common pitfalls
A few mistakes recur often enough to call out directly.
Setting control limits too tight produces alert fatigue, where the team stops trusting the monitoring system because it cries wolf every week, and genuine issues get lost in the noise. Setting limits too loose has the opposite problem: real drift accumulates for months before anyone notices, by which point the fix is expensive and the damage to the business metric has already compounded. Recalibrating limits periodically, and whenever the model itself is retrained, keeps them meaningful.
Monitoring only in aggregate is another common gap. A regional or category level breakdown, as in the worked example above, is often what turns a vague alert into an actionable one within the hour rather than after days of investigation. Aggregating everything into a single company wide number can average out a serious localized problem until it grows large enough to move the whole business metric, by which point it has usually already caused meaningful damage.
Finally, treating every alert as a model bug rather than first asking whether the world changed is a subtle trap. Retraining a model reflexively in response to what turns out to be a genuine, lasting shift in the underlying relationship is reasonable. Retraining reflexively in response to a temporary data quality issue or a one time event, without first triaging the cause, wastes effort and can bake a temporary anomaly into the model's learned behavior going forward.
Looking ahead
Monitoring closes the loop on the technical lifecycle of a model: hypothesis, measurement, experiment, scaling, baseline, and now ongoing observation against that baseline. The remaining question in this series is not technical at all. Once a model is trusted enough to run unattended between alerts, deciding who is accountable for its decisions, what oversight looks like, and where the limits of automated decision making should sit becomes the pressing concern, which is exactly where the next post on ethics and governance picks up.