Measurement variables

In a previous post, the process of data science and forming an hypothesis is discussed. A hypothesis is the relevant to align a business objective to a data science problem. The hypothesis provides a “big-picture” view of the issues which need to considered in further steps of addressing a data science problem.

The problem being considered is insurance fraud, and a good hypothesis for success could be “misrepresentation is different from intentional damage”. This hypothesis attempts to differentiate between misrepresentation and intentional damage.

Misrepresentiation is said to occur when a claim is made on nonexistent assets

Intentional damage is said to occur when an insured asset is intentionally damaged

The next step after an hypothesis is established is to consider variables or factors affecting the hypothesis.

The null Hypothesis

Let us call our hypothesis “misrepresentation is different from intentional damage” - $H$ for mathematical convenience. This can be a hard thing to determine and we can use ideas from statistical testing to develop a solution. A statistical testing process works by determining an antithesis often called the null hypothesis, i.e., if the antithesis were true the hypothesis under consideration would not be true. An antithesis could be “misrepresentation is indifferentiable from intentional damage”, call this $H_0$.

In a traditional scientific experiment, a statistical experiment would be possible by random assignment to conditions under test. In this scenario, one group of insured would generate misrepresentation whereas another group would generate intentional damage claims. Traditional hypothesis testing would calculate a statistic, say a mean, for data generated from two groups and observe if statistic is significantly different from each other.

$$ Experimental\ question: \underbrace{ \begin{cases} \small H: statistic_{misrepresentation} \neq statistic_{intentional} \\
\small H_0: statistic_{misrepresentation} = statistic_{intentional} \end{cases} \tiny \text{verify truth of both statements}}_ {\text{equality/inequality with an acceptable margin of statistical error}} $$

In this scenario, misrepresentation and intentional damange are not randomly assigned or generated from insured parties. In fact, it would be facetious to conduct an experiment to study the problem at hand. Such a problem falls under the umbrella of a natural experiment or observational study depending on the circles you are in.

In an observational study the assignment of population to groups or conditions of the experiment are outside the investigator’s purview. A hypothesis such as “smoking causes cancer” or “video games cause violence” [[1]] is harder to perform in a pure scientific manner. In fact, the earlier position on video games by [[1]] has been attributed to priming by [[3]] and the jury could still be out on this since we cannot guarantee homogenity of the sample in testing for observed effects. In such scenarios the best we can do are observational studies to gain more information about our hypothesis.

What are Measurement Variables (aka Direct Factors)?

In order to perform a scientific study, a data scientist should start by picking up on signals of misrepresentation and intentional damage. These signals are often referred to as measurement variables for modeling. The model of choice for such a problem is a discriminative model, i.e., a model discriminating fraud of misrepresentation and intentional damage. In the old but popular example of discriminating the iris species [[2]], the petal length/width and sepal length/width provided sufficient measurement variables for discrimination of the species using linear functions. In this iris analysis, the experiment was natural, i.e., not in the control of an experimenter.

The term natural means the experimenter did not genetically modify the species to show variations, the variation in the species was naturally selected. On the other hand, in cases such as experiments with fruit flies (available at Indiana University for research), a scientist would study the species by “knocking out genes” or “inducing variations” creating a controlled experiment. The key in either case would be understanding the factors or measurement variables for the hypothesis under study.

A natural/observational experiment is a useful alternative when a controlled experiment cannot be undertaken like the insurance example. It is important to note that a natural experiment can also have issues regarding confounding variables and bias which potentially invalidate the experiment.

A confound (or confounding variable) can be defined as a factor which could directly or indirectly affect the response variable when considering a direct measurement. Let’s take a concrete example here to understand this concept. Assume a scout is looking for talent in basketball (or a VC firm is scouting for investment, the analogy is similar). The scout assesses the talent using a few metrics such as average points per game, assists for offense and rebounds, block, steals for defense. There are other aspects (or confounds) which come into the purview of a scout, such as medical history and stability/improvement of stats because these indicate the progression of a player and future outcomes. In many cases, a confound plays a large role. For example, a player with a debilitating shoulder injury could be a red flag since the future outcome could be weaker with a higher probability. The difficulty would be in ascertaining confounds for the hypothesis under study, and requires understanding the true nature of the effect a confound has on the hypothesis. A targeted interview with an expert (such as claims investigator for insurance or talent scout for sports) is a valuable tool in a data scientists arsenal to understand the factors and confounds which should be considered as data to be included in a model. An interview provides the intuition or priors in a bayesian context for data gathering and evaluation.

A variable or factor discriminating misrepresentation from intentional damage could be identified based on multiple perspectives. Personally, I choose the word perspective as a line of attack/strategy to understand the contributing factors from first principles. This is a preferred approach, in my opinion, to throwing the kitchen sink at a dataset.

Historical variables

Historical variables can be obtained from similar category of claims in the past. They are useful in understanding patterns of normal insurance claims and misrepresentation. Cost per type of damage could be a general factor to monitor, which needs categorizing types of damage available in historical data. In many cases, the insurance system would place restrictions on type of damages covered and bundle similar damages under a large umbrella (because its easier to deal with one type and have a single process). For example flooding could be due to natural events like weather (rain, storm, waves, etc) or a pipe breaking due to stress or damage. Classifying the category at the right level is important in order to provide models the right level of information, not focusing on data driven approaches when collecting data can misclassify labels by not having appropriate levels for a category losing a lot of context.

Textual variables

Textual variables can be obtained from an insurance claim which asks pointed questions to a claimant. Many of the responses to the questions can be free form text or speech which allow representation of the situation in the claim. A misrepresented claim can potentially have signals in the text to describe the situation. Simple constructs would be overuse of certain elements to provide validity to the claim. A speech pattern can have inflection when misrepresenting facts which can be captured by a model.

Another common pattern to obtain signals be asking the same question with a different phrase. Text or speech patterns for both questions should ideally be the similar and a measure of dissimilarity can be used by a model to discriminate between misrepresentation and intentional damage. The details of spacing between the questions and phrasing are experimental variables at the hands of the data scientist to gather useful signals.

Social variables can be obtained from aspects of social interaction such as association to similar groups, participation in similar events or mining social media sites such as Facebook, Twitter, Snapchat etc. The usage of social variables stem from the phrase - “neurons that are fire together wire together” implying that if there is a person who filed a claim with misrepresentation or intentional damage another person could be correlated to do so through social bonds.

Personally, I am not a proponent of using social variables but in some cases they can provide useful information akin to a prior for the model. A data scientist needs to be careful in ensuring the prior or social variables can be overcome by evidence in either direction.

Economic variables

Economic variables capture the financial circumstances surrounding a claim and the claimant. Signals here include the size of the claim relative to the policy premium, the gap between repair and replacement cost for the damaged asset, and whether a claimant has filed multiple recent claims across policies. A claimant under financial stress, evidenced by late premium payments, a recent change in coverage level, or a public record such as a bankruptcy filing, has a different economic incentive profile than one who is not under such stress. Regional economic conditions matter too: claim rates in a zip code can rise during a local downturn or after a large employer lays off workers, and a model that ignores this context risks confusing a broad economic trend for individual misrepresentation.

Economic variables are powerful because they speak to motive, but they should be combined carefully with the historical, textual, and social variables above rather than used alone, since financial hardship by itself is not evidence of fraud.

Identifying measurement variables

The previous section outlined where measurement variables come from: history, text, social context, and economics. Having a long list of candidate variables is not the same as having a useful model. The next step is to identify which of these candidates actually help separate misrepresentation from intentional damage, and to do that a data scientist needs a way to measure how informative a variable is before committing to it. Two complementary tools for this are correlation and separation of classes.

Correlation

Correlation measures how strongly a candidate variable moves together with the outcome. For a continuous variable such as claim to premium ratio, the Pearson correlation coefficient $r$ between the variable and a numeric encoding of the outcome gives a first read on whether the variable carries signal.

$$ r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}} $$

For a variable like whether a claimant has a prior claim, which is a yes or no signal, against a binary outcome, a point biserial correlation or a simple comparison of proportions across the two claim types serves the same purpose.

A high correlation is a useful filter, not proof that a variable belongs in the model. As discussed earlier, a confound can produce a correlation between a variable and the outcome without the variable having any direct bearing on misrepresentation or intentional damage. Zip code level economic conditions correlating with claim type is a good example: the correlation could be picking up the effect of the underlying economic variable rather than the zip code itself. Correlation should be treated as a first pass for narrowing a long list of candidates, with the same confound and domain review from the discussion above used to decide what survives.

Separation of classes

Correlation is a useful filter but it only tells part of the story, especially when a variable does not move in a straight line with the outcome. Separation of classes asks a more direct question: if this variable is plotted for misrepresentation claims and for intentional damage claims side by side, how much do the two distributions overlap? The iris example from earlier in this post is exactly this idea: petal length by itself separates one species cleanly from the other two, which is why it works well as a measurement variable even without computing a correlation coefficient.

A simple way to quantify separation for a numeric variable is the standardized difference between the two class means.

$$ d = \frac{\bar{x}_{misrepresentation} - \bar{x}_{intentional}} {s_{pooled}} $$

Here $s_{pooled}$ is the pooled standard deviation across both groups. A larger $d$ means less overlap between the classes and a more useful variable on its own. A variable that does not separate classes well by itself can still be useful in combination with others, which is why a full model considers combinations of measurement variables rather than judging each one in isolation.

References

[1] Engelhardt, C.R. et al. 2011. This is your brain on violent video games: Neural desensitization to violence predicts increased aggression following violent video game exposure. Journal of Experimental Social Psychology. 47, 5 (2011), 1033–1036.

[2] Fisher, R.A. 1936. The use of multiple measurements in taxonomic problems. Annals of eugenics. 7, 2 (1936), 179–188.

[3] Kühn, S. et al. 2019. Does playing violent video games cause aggression? A longitudinal intervention study. Molecular psychiatry. 24, 8 (2019), 1220–1234.

Back to posts