Generalizing or scaling the experiment (1 to n)

The last two posts in this series covered modeling of patterns for prediction, first through numerical analysis for error reduction and then through qualitative modeling. By the end of that stage, mysurance had something concrete to show for its efforts: a model that discriminated misrepresentation from intentional damage well enough to act on, validated against a held out sample of the population it was built on.

This post asks the question that comes right after that validation: now what? A model validated on one product line, one region, or one slice of claims is not the same thing as a model that works everywhere the business needs it to work. Earlier in this series, the post on experimental design described the journey from having no working approach to having one, what I called the zero to one framing, borrowing the phrase from the way founders and economists talk about going from nothing to a first working version. This post is the mirror image of that one. Having gotten to one validated result, the task now is to go from one to n: taking a finding that held up under controlled conditions and asking whether, and how, it holds up everywhere else.

1. Hypothesis
2. Measurement variables
3. Latent or unobservable factors
4. Experimental design (0 to 1)
   1. Controlling other factors to observe primary effect.
5. Collection and analysis of data for pattern discovery
   1. Hypothesis driven Exploration
6. Modeling of patterns for prediction
   1. Numerical Analysis for error reduction
   2. Qualitative modeling
7. Generalizing or scaling the experiment (1 to n) (discussed here)
8. Establishing a baseline
9. Monitoring through controls and baselines
10. Ethics and governance

From zero to one, to one to n

It is worth being precise about what changes between these two stages, because the two are governed by different failure modes and different skills.

Getting from zero to one is a causal problem. The question at that stage was whether an effect exists at all, under conditions the data scientist could control or at least carefully observe. In the mysurance example, that meant isolating misrepresentation from intentional damage using a curated set of measurement variables, controlling for confounds, and checking that the discriminative model performed better than chance on held out data from that same setting.

Getting from one to n is a transportability problem. The question is no longer whether the effect exists, it is whether the same effect, measured the same way, continues to hold when the population, the operating conditions, or the scale change. A model can be entirely correct about the population it was validated on and still be wrong, in a precise and predictable sense, about a population it has never seen.

This distinction matters because the instinct after a successful zero to one result is to declare victory and roll the result out broadly. In my experience that instinct is usually premature. A result that holds in one controlled setting is a necessary condition for scaling, not a sufficient one.

External validity: does the effect travel

Statisticians and experimental psychologists distinguish between internal validity, whether an observed effect is really caused by what you think caused it within the sample you studied, and external validity, whether that same effect generalizes to other people, places, times, or conditions. The zero to one post in this series was largely about internal validity: controlling other factors so the primary effect could be observed cleanly. Scaling the experiment is squarely about external validity.

External validity can fail for reasons that have nothing to do with a flawed original experiment. A few common ones that show up repeatedly in industry settings:

• Selection effects. The population used for validation was not a random sample of the population the model will eventually serve. It may have been the most instrumented segment, the one with the cleanest historical labels, or simply the one that was easiest to get data science attention for first.
• Interaction effects. The measurement variables that discriminate fraud in one context may depend on other factors that differ elsewhere. A textual variable built around phrasing patterns in English claim narratives, for instance, may carry very different signal in a region where claims are filed in a different language or through a different channel.
• Period effects. The original result was observed during a particular window of time, under particular economic or seasonal conditions, that may not persist.
• Base rate shifts. The prevalence of the effect being modeled, fraud rate in this case, can differ substantially across segments even when the underlying mechanism is the same.

None of these threats are hypothetical. They are the ordinary cost of moving from a controlled setting to the messier setting of full deployment, and a data scientist scaling an experiment should treat checking for them as part of the job, not as an afterthought.

Simpson's paradox: when aggregation reverses the effect

One of the sharpest illustrations of why external validity cannot be assumed is Simpson's paradox, the phenomenon where a trend that appears in several different groups of data disappears, or even reverses direction, when those groups are combined.

Consider a simplified version of a check mysurance would need to run before scaling a fraud model. Suppose the model flags claims as high risk or low risk, and we look at the confirmed fraud rate within each flagged group, separately for two product lines.

$$ \begin{array}{c|cc|cc} & \text{Auto} & & \text{Homeowners} & \\ & \text{Flagged} & \text{Not flagged} & \text{Flagged} & \text{Not flagged} \\ \hline \text{Confirmed fraud rate} & 40\% & 10\% & 25\% & 5\% \end{array} $$

Within each product line the model is doing exactly what we want, the flagged group has a noticeably higher fraud rate than the unflagged group. Now suppose auto claims are large in volume but low in fraud prevalence overall, while homeowners claims are smaller in volume but the flagged subgroup happens to be a much larger share of all homeowners claims. Depending on how the two groups are weighted when combined, the aggregate comparison of flagged versus unflagged can shrink toward parity, or in more extreme configurations of the underlying counts, invert entirely, showing the unflagged group with a higher rate than the flagged group once the two lines are pooled.

The mechanism is not mysterious once written out. If $p_i$ is the fraud rate within flagged claims for line $i$, and $w_i$ is the share of flagged claims coming from line $i$, the aggregate flagged rate is a weighted average $\sum_i w_i p_i$, and the same holds for the unflagged rate with its own weights. When the mix of volume across lines, $w_i$, differs between the flagged and unflagged populations, the two weighted averages are not simply summarizing the same underlying relationship, they are summarizing different mixtures of it. A model can look worse, better, or reversed in aggregate purely as an artifact of how the subgroups happen to be weighted, with no change at all in how well it discriminates fraud within any single product line.

The practical lesson for scaling an experiment is this: never evaluate a model's effect only at the aggregate level once more than one segment is in play, and never assume a subgroup effect from an aggregate one. Both directions of inference can fail. The only way to know whether an effect holds is to check it, stratified, in each subgroup you plan to deploy into, and to keep checking as new subgroups are added.

Infrastructure and operational concerns that only appear at scale

External validity is a statistical concern. Scaling also introduces a second, entirely separate category of concern that has nothing to do with whether the effect is real, engineering and operations. A model that is statistically sound can still fail in production for reasons that a zero to one validation, run on a modest sample with generous turnaround time, would never surface.

Latency

A model validated in a notebook or batch job might score a few thousand claims overnight without anyone noticing how long it took. The same model, deployed to score every incoming claim in real time so an adjuster can see a risk flag the moment a claim is filed, now has a latency budget measured in tens or low hundreds of milliseconds. Feature computation that pulls from several systems, joins historical variables, and runs text processing on a claim narrative can easily blow past that budget once it needs to happen synchronously, at full volume, for every product line at once.

Cost per prediction

A model that costs a few cents to score during validation, perhaps because it calls out to a slower service, an external data enrichment API, or a large text model, can become a meaningfully large recurring cost once it is scoring millions of claims a month rather than a few thousand in a pilot. Cost per prediction is rarely examined during the zero to one stage because the volumes are too small for it to matter. It has to be examined before deciding to scale, because it can change which architecture is even viable at full volume.

Data pipeline reliability

A pilot typically runs on a carefully prepared, relatively static extract of data. A model in production depends on a live pipeline: claims systems that occasionally go down, upstream schema changes that break a feature silently, fields that are populated inconsistently across product lines or regions, and missing data that behaves differently at 2am on a holiday than during a Tuesday afternoon. None of this is visible in a one time validation. It becomes visible, often painfully, the first time an upstream team changes a field definition without telling anyone downstream.

These operational concerns interact with the external validity concerns discussed above. A pipeline outage in one region during a rollout can look, statistically, exactly like a real drop in effect size in that region if the two are not distinguished carefully.

A staged rollout instead of an all at once launch

Given both categories of risk, statistical and operational, the sensible way to scale an experiment is gradually, not all at once. A staged rollout expands coverage, whether by product line, region, or customer segment, in a sequence of small steps, checking at each step that the effect still holds and that the operational characteristics remain acceptable before expanding further.

A reasonable staging plan follows a few principles:

• Start closest to the validated setting. Expand first into the segment most similar to the one the model was originally validated on. This maximizes the chance that external validity holds and isolates any new failures to genuinely new factors rather than confounding several unknowns at once.
• Expand one dimension at a time where possible. Adding a new region within a familiar product line, or a new product line within a familiar region, is easier to diagnose than adding both simultaneously.
• Recompute the effect within each newly added segment before adding the next one. This is exactly the stratified check Simpson's paradox motivates, an aggregate metric across all segments launched so far is not sufficient, each new segment needs its own check.
• Define a rollback criterion in advance. Decide, before launching a stage, what evidence would cause you to pause or roll back that stage. Deciding this after seeing disappointing results invites motivated reasoning about why the result does not really count.
• Budget time for the operational concerns to surface. Latency and pipeline issues often only appear under sustained full volume, not in the first hour of a new segment going live.

A staged rollout is slower than an all at once launch, deliberately so. The cost of that slowness is measured in delayed value from segments that would have been fine. The cost of skipping it is measured in decisions made on a model that silently stopped working the moment it left the setting it was validated in, which is a considerably worse failure to discover after the fact.

Worked example: scaling mysurance's fraud model across product lines and regions

Return to mysurance. The fraud detection approach discussed earlier in this series, discriminating misrepresentation from intentional damage using historical, textual, and social measurement variables, was validated on homeowners claims filed in a single region. The model performed well there: a discrimination threshold was set, adjusters using it caught more confirmed fraud with fewer false accusations than the previous manual process, and leadership, understandably, wants it everywhere: auto, renters, and commercial lines, in every region mysurance operates in.

Before treating this as a rollout rather than a rewrite, a few checks need to happen.

Are the measurement variables even available

The textual variables that were useful for homeowners claims came from free form narrative fields in the claim intake form. Does the auto claims intake form even collect a comparable narrative field, and if it does, are claimants answering it the same way? Historical variables built around categories of homeowners damage, water damage versus fire versus theft, do not map cleanly onto auto claims categories, collision versus comprehensive versus liability. A variable that does not exist, or means something different, in the new product line cannot simply be plugged into the same model.

Does the base rate match

Fraud prevalence is unlikely to be identical across product lines and regions. A threshold tuned to flag the top few percent of homeowners claims by predicted risk implicitly assumes a base rate of misrepresentation similar to what was observed in the validation sample. If commercial claims have a meaningfully different base rate, the same threshold can produce a very different mix of true positives to false positives, since the positive predictive value of a fixed threshold is not fixed, it moves with prevalence. This is a direct, practical consequence of the base rate shift concern raised earlier under external validity, and it means thresholds, not just the underlying model, need to be recalibrated per segment rather than copied over.

Check for Simpson's paradox across segments

Once auto, homeowners, renters, and commercial are all live, mysurance should not be satisfied with a single company wide fraud catch rate as the headline metric. As the numeric illustration earlier in this post showed, a healthy looking aggregate number can be masking a product line, or a region within a product line, where the model is performing poorly or has effectively stopped discriminating fraud at all, simply because that line's volume and mix are different from the others being pooled into the same number. The fix is the same discipline mentioned above: report and monitor the effect stratified by segment, not only in aggregate.

Check the infrastructure per segment

Commercial claims at mysurance, in this scenario, are far higher in dollar value and lower in volume than personal auto claims, and are handled by a different claims processing system with its own latency and data quality characteristics. A feature pipeline built for high volume, low latency personal auto scoring will not automatically behave the same way against the commercial claims system, and the cost per prediction of any component that calls an external service will scale differently against a system with a different transaction volume profile.

Roll out in stages, not all at once

Given all of the above, the sensible path for mysurance is to expand into the product line and region closest to the original validated setting first, perhaps renters insurance in the same region, since renters claims share more structural similarity with homeowners claims than auto or commercial do. Confirm the effect holds there, stratified, and that latency and pipeline reliability are acceptable at that volume. Only then expand into auto, where measurement variables need the most rework, and commercial last, where volume is lowest but the operational profile differs the most. At each stage, thresholds get recalibrated to the local base rate rather than inherited wholesale from the original validation.

None of this changes the underlying hypothesis mysurance started with, that misrepresentation is different from intentional damage, and it does not mean the original model was wrong. It means that a result validated at one setting has to earn its way, deliberately and empirically, to many.

Once a model has been scaled and is running across the segments it needs to cover, the natural next question is how to know, on an ongoing basis, whether it is still working as well as it was on the day it was validated. That is the subject of the next post in this series, establishing a baseline.

Back to posts