Latent or unobservable factors

In a previous post, measurement variables are discussed. A measurement variable is something a data scientist can point to in a dataset and say “this column is what I collected, and this is exactly what it represents”. Petal length, points per game, the words in a claim transcript, these are all direct measurements. They exist as numbers or text the moment we look for them.

Not every quantity that matters to a hypothesis is like that. Some of the most important drivers of an outcome cannot be measured directly at all. We cannot open up a customer and read off a number for how frustrated they are. We cannot query a database column called intent_to_leave. These are latent, or unobservable, factors, and this post is about how a data scientist works with them anyway.

1. Hypothesis
2. Measurement variables
3. Latent or unobservable factors (discussed here)
4. Experimental design (0 to 1)
   1. Controlling other factors to observe primary effect.
5. Collection and analysis of data for pattern discovery
   1. Hypothesis driven Exploration
6. Modeling of patterns for prediction
   1. Numerical Analysis for error reduction
   2. Qualitative modeling
7. Generalizing or scaling the experiment (1 to n)
8. Establishing a baseline
9. Monitoring through controls and baselines
10. Ethics and governance

Measurement variables vs latent variables

A measurement variable is observed. A latent variable is inferred. That one distinction changes almost everything about how a data scientist has to treat it.

A measurement variable is a quantity recorded directly in the data, its value is a fact about the world once it is collected.

A latent variable is a quantity that is believed to exist and to influence the outcome, but whose value can only be estimated through other, observable variables that are affected by it.

Consider the insurance fraud hypothesis from earlier in this series, “misrepresentation is different from intentional damage”. The claim amount, the type of damage, the time between policy purchase and claim, these are measurements. But something like “the claimant's true intent” is latent. Nobody fills out a form with their intent as a numeric field. We only ever see behavior that intent produces: inconsistent statements, unusual timing, patterns in phrasing. The intent itself sits behind a curtain, and the measurements are the only light getting through.

This is not a minor technicality. A huge share of the variables data scientists actually care about, trust, satisfaction, skill, risk tolerance, engagement, are latent by nature. They are the underlying constructs a business actually wants to reason about. The measured columns in a dataset are almost always stand ins, chosen because the real thing cannot be recorded directly.

Proxies and indicators

Since a latent variable cannot be observed, a data scientist works with a proxy (also called an indicator) instead: an observable variable that is believed to move together with the latent factor closely enough to stand in for it.

A good proxy has two properties worth naming explicitly.

It should be caused by the latent factor

The causal arrow should point from the latent variable to the proxy, not the other way around, and ideally not from some third, unrelated cause to both. If frustration causes a customer to contact support, then support contact is a reasonable proxy for frustration. If instead support contact were driven mostly by something unrelated, like a product feature that happens to confuse everyone equally regardless of how satisfied they are, it would be a weaker proxy.

It should vary with the strength of the factor

A proxy that only reacts once frustration is already extreme is much less useful than one that tracks the buildup. Ideally a proxy responds gradually and monotonically as the underlying latent factor increases, so that changes in the proxy carry information about changes in the latent factor, not just its presence or absence.

In practice a data scientist rarely relies on a single proxy. Multiple proxies, each imperfect on its own, are combined because their combination reduces the noise that any one proxy carries on its own. This is exactly the idea behind factor analysis and latent variable models, covered next.

Factor analysis and latent variable models, the general idea

Factor analysis is a family of techniques built around a simple premise. If several observed variables are all being pulled by the same unobserved factor, they should correlate with each other more than they would by chance, and that shared correlation can be used to estimate the factor itself.

Formally, an observed variable $x_i$ is modeled as a function of one or more latent factors $f$ plus some variable specific noise $\epsilon_i$ that is not shared with any other observed variable.

$$ x_i = \lambda_i f + \epsilon_i $$

The term $\lambda_i$ is called a factor loading. Intuitively, it is just a weight describing how strongly and in what direction the latent factor shows up in that particular observed variable. A large positive loading means the proxy moves strongly in the same direction as the factor. A loading near zero means that proxy barely reflects the factor at all, its variation is mostly noise or something else entirely. A negative loading means the proxy moves opposite to the factor, which is perfectly fine as long as it is accounted for, a rising value on that proxy would then suggest a falling value of the underlying factor.

None of this requires solving the equations by hand to be useful. The important intuition to carry forward is this: a factor loading tells you how much to trust a given proxy as a window into the latent variable, and a factor analysis or latent variable model is really just a principled way of asking several proxies to vote, weighted by how much each one actually reflects the thing you care about, and combining their votes into a single estimated score for the latent factor.

The same idea appears under different names depending on the field. Psychometrics calls it factor analysis when scoring traits like anxiety or extraversion from questionnaire items. Structural equation modeling generalizes this further by allowing latent factors to also predict other latent factors. In machine learning, a similar spirit shows up in techniques that compress many correlated features into a smaller number of components, although the goal there is often dimensionality reduction rather than recovering a specific interpretable construct.

Proxy failure: when the stand in stops standing in

A proxy is only useful as long as it keeps tracking the latent factor it was chosen for. This can break in a few distinct ways, and a data scientist should be able to name which one is happening rather than just noticing that a model has degraded.

The proxy was never a good match

Sometimes a proxy is picked for convenience rather than fit, it is the column that happened to be easy to collect, not the one that actually tracks the construct. A classic example outside data science is using standardized test scores as a proxy for a student's aptitude, when in practice the score also reflects test anxiety, quality of the school's test preparation, and familiarity with the test format. The proxy captures a mixture of several things, only one of which is the intended latent factor.

The relationship between proxy and factor drifts over time

A proxy that worked well when it was chosen can decay as behavior around it changes. If people learn that a certain observable behavior is being watched, they may adjust the behavior without any change in the underlying latent factor. This is sometimes discussed under Goodhart's law, once a measure becomes a target, it tends to stop being a good measure.

The proxy gets gamed or manipulated

Closely related, if the proxy itself has consequences (a support agent is scored on how many tickets they close, so closure counts stop meaning what they used to), the proxy can be actively distorted by the people generating it.

The common thread in all three failure modes is the same: the proxy and the latent factor quietly come apart, while the model or the business process keeps trusting the proxy as if nothing changed. Watching for this divergence, not just monitoring the proxy in isolation, is a recurring theme that comes back later in this series when we discuss monitoring through controls and baselines.

Worked example: predicting churn for a streaming service

Consider a media streaming product, and the business question is churn: which subscribers are likely to cancel next month. The hypothesis a data scientist starts with might be something like “subscribers who are frustrated with the product, or who have already mentally decided to leave, cancel at a much higher rate than subscribers who have not”.

The latent factor here could be described a couple of different ways, and it is worth being precise about which one is being modeled, because they are related but not identical constructs.

Customer frustration is a latent state reflecting dissatisfaction with the product experience.

Intent to leave is a latent state reflecting an active decision or leaning toward cancellation, which frustration often contributes to but does not fully determine.

Neither can be read off directly. Nobody logs into their account and sets a frustration_level field. What we do have is a set of behaviors that frustration and intent to leave are believed to cause.

Candidate proxies

Support ticket sentiment

When a subscriber contacts support, the tone and content of that interaction (extracted with a sentiment model, or something simpler like counting complaint keywords) is a reasonably direct behavioral trace of frustration. It has a clear causal story, frustration leads people to complain, so this proxy is well aligned to the factor. The tradeoff is coverage: only a minority of subscribers ever contact support at all, so this proxy is sparse. For most of the subscriber base it will simply be missing.

Drop in login frequency

A subscriber who used to log in four times a week and now logs in once is showing disengagement. This proxy has excellent coverage since it is recorded for every account automatically, but its causal story is messier. Login frequency drops for plenty of reasons that have nothing to do with frustration or intent to leave, a vacation, a busier month at work, a household member who used to control the shared account moving out. On its own this proxy is noisy.

Reduced watch time

Total minutes watched per week captures actual product usage more directly than login frequency (a subscriber could log in out of habit and watch almost nothing). Watch time shares some of the same confounds as login frequency though, seasonal patterns, competing shows on other platforms during a slow content month for this one, a household simply watching less television in the summer.

Validating a proxy before trusting it

A data scientist should not simply assume any of these three tracks the latent factor well just because it sounds plausible. A few concrete checks are worth running before building a model around a proxy.

Check against a known outcome

Churn itself, the eventual cancellation, is an observed event, even though the reasons behind it are latent. So the most direct validation is to check whether the proxy actually predicts churn among subscribers who have already cancelled. If subscribers who cancelled showed a meaningfully larger drop in login frequency in the weeks before cancelling compared to subscribers who did not cancel, that is evidence the proxy is correlated with the thing that matters. This does not prove the proxy measures frustration specifically, only that it correlates with the downstream outcome we ultimately care about, but it is a necessary condition for a proxy to be useful.

Check consistency across proxies

If frustration is real and driving multiple behaviors, the proxies should correlate with each other more than chance would suggest, this is the same logic behind factor analysis discussed earlier. A subscriber showing negative support sentiment should also be somewhat more likely to show a drop in watch time, not because one causes the other directly, but because both are downstream of the same latent state. If support sentiment and watch time show essentially no relationship to each other at all, that is a signal at least one of them is dominated by noise or by a different, unrelated cause, rather than by the latent factor we are trying to capture.

Check for plausible confounds

Before trusting a drop in watch time as a proxy for frustration, a data scientist should ask what else could produce that same drop. Seasonality is an obvious one, comparing the same subscriber's watch time to the same period a year earlier, rather than only to last month, helps separate a seasonal dip from a genuine decline. Household composition changes are harder to observe directly but can sometimes be inferred from account level signals such as a drop in the number of distinct profiles being used.

Combine proxies with loadings rather than picking one

Rather than choosing a single winner among the three candidate proxies, the practical approach is to combine them, weighting each one by how strongly and reliably it tracks churn once the confounds above are accounted for. This is the applied version of the factor loading idea: support ticket sentiment might get a larger weight because its causal story is cleanest, even though it is sparse, while login frequency and watch time contribute smaller weights individually but add coverage and stability because nearly every subscriber has them. The combined score is treated as an estimate of the latent “intent to leave” factor, and it is this combined score, not any single proxy, that feeds the churn model.

Revalidate periodically

Because proxies can drift or be gamed, as discussed above, the relationship between each proxy and actual churn should be rechecked on a schedule rather than assumed to hold forever. If the streaming product changes its support ticket workflow, for instance moving from live chat to a bot for a first response, the sentiment expressed in the resulting transcripts can shift in ways that have nothing to do with subscriber frustration, and the proxy's weight in the combined score needs to be revisited.

Why this matters for what comes next

Getting the latent factor and its proxies right at this stage has direct consequences for the steps ahead in this series. An experimental design that intends to move the needle on churn needs a target it can actually observe changing, and that target is usually the combined proxy score, not the true unobservable factor. Getting the proxy selection wrong here means an experiment later could show no measurable effect on the proxy while the true latent factor, if it could be seen, moved substantially, or the reverse, an experiment could appear to move the proxy while doing nothing to the real underlying issue. The next post in this series covers experimental design, and the choice of what to measure as the outcome of an experiment leans directly on the work done here.

Back to posts