Process of Data science - Qualitative modeling

Jun 25, 2021

14 minutes read

Qualitative modeling

The last two posts in this series were both about fitting numbers to data. We built a risk score for hospital readmission and then spent an entire post on numerical analysis, tightening that score by chasing down sources of error. Both posts assumed the same starting point: enough historical, structured, labeled data to hand to an optimizer and let it find a function that predicts an outcome.

This post is a deliberate turn away from that path. Not every pattern worth modeling has the data, the label, or even the appetite for a fitted function behind it. Sometimes the right model is a rubric that a person can read and apply, built from interviews and edge case discussions rather than from a loss function. I want to cover what qualitative modeling means, how it is built, and when it should be chosen over a numeric model instead of being treated as merely a stepping stone toward one.

What qualitative modeling means in a data science context

A model, in the broad sense, is just a formalized structure for turning an observation into a decision. Most of this series has treated that structure as a function learned from data: pick a family of functions, pick a loss, let an optimizer choose parameters that minimize the loss over a training set. That is a fitted model.

A qualitative model is also a formalized structure for turning an observation into a decision, but the structure is authored by a person rather than learned by an optimizer. It is still a model in every meaningful sense: it is explicit, it is applied consistently, and it can be tested and revised. What differs is where the structure comes from. A fitted model compresses a training set into parameters. A qualitative model compresses expert judgment, domain reasoning, and observed edge cases into a set of rules, categories, or a decision procedure that a human can read and apply directly.

It helps to resist a common assumption in data science circles, that qualitative work is a lesser or temporary stand in for a "real" model that will eventually replace it. Sometimes that is true. Often it is not. A rubric that a trained reviewer applies consistently can be more accurate, more explainable, and more adaptable to new situations than a classifier trained on a few thousand historical labels, particularly when the domain keeps shifting underneath the model.

Rule based systems and expert defined heuristics

The oldest and most direct form of qualitative modeling is a rule based system: an explicit set of if then statements written by a person who understands the domain, rather than induced from data. Early expert systems such as MYCIN, built at Stanford in the 1970s to recommend antibiotic therapy from a patient's symptoms and lab results, worked exactly this way. Physicians and researchers encoded their diagnostic reasoning as a network of rules with associated confidence factors, and the system applied those rules to new cases [1].

A heuristic is a compressed piece of expertise: a short rule of thumb that captures a regularity an expert has learned through years of seeing cases, without needing to justify that regularity from first principles every time. "Claims filed within 48 hours of a policy change deserve a second look" is a heuristic. So is "a support ticket with more than three exclamation points and a threat to cancel usually needs a human, not a canned response." Neither of these came from a regression. They came from someone noticing a pattern often enough to trust it.

Rule based systems have real strengths. They are transparent: anyone can read the rule and understand exactly why a decision was made. They do not need a large labeled dataset to get started, only a person willing to sit down and write out what they already know. And they can encode reasoning that is hard to express as a function of a handful of numeric features, such as "unless the customer has already been warned twice this month."

They also have well known pitfalls. Rules written by hand are brittle: they cover the cases the author thought of, and silently fail on cases the author did not think of. They do not automatically improve as more data arrives, someone has to notice the gap and add or revise a rule. And as the rule set grows, rules can interact in ways nobody intended, producing contradictions or blind spots that only show up during an audit. A rule based system is a living document, not a one time deliverable, and it needs the same discipline around versioning and review that a numeric model needs around retraining.

Thematic coding of text and qualitative feedback

A second major form of qualitative modeling shows up whenever the raw material is text: open ended survey responses, support tickets, interview transcripts, free form complaint fields on a claim form. The technique for turning that unstructured feedback into a usable model is called thematic coding, and it comes out of qualitative research methodology rather than out of natural language processing [2].

Thematic coding works by reading through a sample of the text and identifying recurring themes, meaningful patterns of content that matter for the question at hand. Those themes get organized into a codebook: a short, explicit definition for each theme along with example passages that illustrate it. A second pass then applies the codebook to the full set of text, tagging each passage with the themes it matches.

This is deliberately not the same exercise as fitting a bag of words classifier or a topic model. A statistical topic model finds groups of words that co occur; it has no notion of what those words mean to the person who wrote them. Thematic coding asks a person, or a small team of trained coders, to interpret meaning, which lets it capture things a word co occurrence pattern cannot, such as sarcasm, a complaint framed as a compliment, or a theme that only ever appears combined with another theme in a way that changes its meaning.

Because thematic coding depends on human interpretation, it needs a check that a fitted model does not: agreement between coders. If two people code the same passages and frequently disagree on which themes apply, the codebook is not well defined enough to trust yet. A standard way to quantify this is Cohen's kappa, which measures agreement between two raters while correcting for the agreement expected by chance alone [3].

$$ \kappa = \frac{p_o - p_e}{1 - p_e} $$

Here $p_o$ is the observed proportion of passages where the two coders agree, and $p_e$ is the proportion of agreement expected if both coders were assigning themes at random, given how often each coder used each theme. A $\kappa$ near zero means the codebook is doing no better than chance at capturing a shared understanding, and the definitions need to be sharpened before the coding results can be trusted as a model of anything.

When qualitative judgment should be preferred over a fitted numeric model

Given a choice, when should a data scientist reach for a rubric instead of reaching for training data and an optimizer? Three situations come up often enough to be worth naming explicitly.

Too little data to fit a reliable model

A fitted model needs enough examples to estimate its parameters with acceptable uncertainty. Roughly speaking, the standard error of an estimate shrinks with the square root of the sample size,

$$ SE \propto \frac{1}{\sqrt{n}} $$

which means a model trained on a few dozen labeled examples carries enough uncertainty in its parameters that its predictions on new cases are little better than a guess, even if the fitted curve looks reasonable on the training data itself. Early on, before an outcome has happened often enough to build a dataset, a qualitative rubric written from expert reasoning is simply the more honest model. It does not pretend to a precision the data cannot support, and it can be applied from day one rather than waiting for enough incidents to accumulate.

Decisions that need to be explainable in plain language

A logistic regression coefficient or a gradient boosted tree's feature importance is a perfectly good explanation to another data scientist. It is a poor explanation to a customer who was denied a refund, a patient who was flagged for a follow up visit, or a moderator's supervisor asking why a post was removed. A rubric written in plain language, "this comment was removed because it named a private individual alongside an accusation with no supporting evidence," is something a non technical stakeholder can read, agree with or dispute, and act on. When the audience for the reasoning behind a decision is not a data scientist, a qualitative rubric usually communicates better than a numeric score ever will, regardless of how accurate the score is.

The cost of a wrong automated decision is high

Every model, qualitative or fitted, makes mistakes. The question is who catches the mistake and how expensive it is when nobody does. A fitted model applied automatically at scale will apply its errors at the same scale, all at once, with no person in the loop to catch an unusual case. When a wrong decision is expensive enough, in money, in harm to a person, or in reputation, it is often safer to route the decision through a human reviewing a rubric than through a classifier acting alone, even if the classifier's average accuracy looks good on paper. Average accuracy is a statement about the typical case; a costly wrong decision is usually the atypical case, which is exactly where a fitted model's guarantees are weakest and a trained reviewer's judgment is strongest.

Qualitative and quantitative modeling working together

None of this means qualitative and quantitative modeling are a permanent either or choice. In practice the two feed each other, and the direction that matters most is the one that gets skipped when a team jumps straight to a classifier: a qualitative rubric, built from expert interviews and edge case discussion, can become the label set that a later numeric model learns from.

Consider what a rubric actually produces once it has been applied consistently for a while: a growing set of cases, each with an input and a decision that a trained person made by following an explicit, readable procedure. That is precisely the shape of a labeled training set. The rubric did the hard work that a data science project usually struggles with most, namely deciding what the label should mean and producing it consistently, before a single row of training data was collected for a model.

This also changes what the eventual numeric model is for. It is not there to replace human judgment with something smarter. It is there to apply an already validated form of judgment faster and at a volume no review team could sustain, while the rubric remains the source of truth for what a correct decision looks like and the reference for auditing the model's mistakes.

Worked example: a content moderation policy for a small online community

Suppose a small online community, a few thousand active members discussing a hobby, needs a moderation policy. The team could jump straight to training a classifier on some off the shelf toxicity dataset. This is exactly where the hospital readmission model from two posts ago and this problem part ways in approach, and the difference is worth sitting with.

The readmission model had a mature, structured, richly labeled history behind it: years of admissions, discharges, diagnoses, and a clear, already agreed upon outcome to predict. The measurement variables and latent factors had been debated by clinicians for decades before a data scientist ever touched the dataset. Fitting a numeric model there was reasonable from day one because the label and the data both already existed at scale.

A brand new community has none of that. There is no historical set of labeled "this post should have been removed" examples, no agreed definition of what counts as harassment in this specific community's culture, and a membership small enough that a handful of misjudged removals could sink trust in the platform entirely, exactly the high cost, low data, needs plain language explanation combination described above. So the team starts qualitative instead.

Building the rubric from moderator interviews

The team interviews the volunteers who have been informally moderating similar communities elsewhere, and asks them to walk through recent edge cases: posts they struggled to make a call on, not the easy ones. From those conversations a first rubric emerges, something like the following.

Remove immediately: direct threats, sharing another member's private information without consent, sexual content involving minors.

Remove after a second opinion from another moderator: personal insults directed at a named member, repeated off topic spam from the same account after a warning.

Leave up but flag for the weekly review meeting: heated disagreement about the hobby itself, sarcasm that could be read as an insult depending on prior context between the two members, criticism of a moderator's own past decision.

Notice what this rubric is doing. It is not trying to reduce every post to a single toxicity number. It is sorting posts into buckets that call for different levels of human attention, which matches how the volunteers already think about the problem, and it is explainable to a member who asks why their post was removed: the moderator can point to the specific line of the rubric that applied.

Stress testing the rubric with edge cases

The team then deliberately tries to break the rubric with harder edge cases: a joke between two long time members that would read as an insult to a newcomer, a critical but well argued post about a rule change, a member venting frustration at the hobby itself rather than at a person. Each case that the rubric handles badly gets a note added, a clarifying sentence or a new bucket, the same way a fitted model gets a new feature when a residual analysis turns up a systematic error. The rubric is a living document in the same sense the rule based systems discussed earlier are, it needs revision as new cases surface, not a one time document handed down from the interviews and left alone.

When and how the rubric could later seed a numeric model

As the community grows and moderators apply the rubric to a rising volume of posts, each decision, the post, the bucket it was sorted into, and which line of the rubric justified it, becomes a labeled example. Once that log reaches a few thousand consistent decisions, made by moderators who agree with each other closely enough to pass an inter rater check similar to the kappa statistic used for thematic coding, the team has something it did not have on day one: a labeled dataset whose label definition is already validated by months of human review.

At that point, training a classifier to handle the high volume, easy buckets, the obvious spam, the clear cut removals, and routing anything the classifier is uncertain about back to a human reviewing the original rubric, is a sound next step. The model is not replacing the rubric, it is automating the part of the rubric's job that has become routine enough to trust to automation, while the rubric keeps its role as the definition of correct behavior and the tool for auditing the model when a member appeals a decision. This is the opposite order from the readmission model, where the numeric fit came first because the label and the data already existed. Here the qualitative rubric had to exist first in order to create a label worth fitting a model to at all.

Looking ahead

Whether a pattern ends up encoded as a fitted numeric model, a rule based rubric, or some combination of the two that hands off routine cases from one to the other, the next question in this series is the same: how does a model that works on a small, carefully observed population get generalized or scaled up to the full population it is meant to serve. That is the subject of the next post.

References

[1] Shortliffe, E.H. 1976. Computer based medical consultations: MYCIN. Elsevier.

[2] Braun, V. and Clarke, V. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology. 3, 2 (2006), 77 to 101.

[3] Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 20, 1 (1960), 37 to 46.

Back to posts