Ethics and governance

In the last post, we discussed monitoring through controls and baselines, the discipline of watching a model after it has shipped so that quiet degradation in accuracy or drift in the underlying population gets caught before it costs real money. Monitoring answers the question "is the model still doing what we built it to do." This closing post asks a different question, one that sits alongside every step covered so far rather than after them: is the model doing the right thing, and who is answerable if it is not.

This is the tenth and final post in the series. Rather than reach for a new domain the way a few of the other posts in this batch have (a checkout flow, a hospital readmission problem, a ride hailing marketplace), this post returns one last time to mysurance, the insurance fraud detection problem that opened the series in the first two posts on hypothesis and measurement variables. Following one problem all the way from a plain hypothesis statement to a governed, production model is the point of ending the series here rather than introducing something new.

Ethics is not a separate step

Numbering this series one through ten invites a reading of data science as a pipeline, hypothesis first, governance last, each step finished before the next begins. That reading is convenient for writing a series of posts, but it is not how the underlying practice actually works. Ethics and governance questions apply at every step already covered, not only at the end. Which hypothesis a team chooses to test is itself a choice with consequences: testing "misrepresentation is different from intentional damage" frames fraud as a classification problem to be automated, and that framing is already a decision about who gets flagged and who does not. Which measurement variables get collected, which factors get scaled from one experiment to a general system, and what gets monitored in production are all decisions that shape who a model helps and who it burdens. Placing governance last in this series gives us the vocabulary to name these concerns precisely, but the concerns themselves have been present since the first post.

Fairness in a model context

Fairness does not reduce to a single number, and a full treatment of the competing formal definitions of fairness in machine learning is well beyond one blog post. For a working data scientist, the practical starting point is a single observation worth internalizing: a model can be statistically accurate overall while still performing noticeably worse for an identifiable subgroup of the population it serves, and this can be true even when no one intended it and no single line of code says to treat that subgroup differently.

The reason this is possible, and not merely a hypothetical, follows directly from how an aggregate accuracy number is constructed. Overall accuracy is a weighted average of accuracy across every group $g$ in the population $G$ the model touches, weighted by how much of the population each group represents.

$$ Accuracy_{overall} = \sum_{g \in G} p(g) \cdot Accuracy_g $$

If a group $g$ makes up a small share $p(g)$ of the population, that group's accuracy can be substantially lower than the overall figure without moving $Accuracy_{overall}$ very much at all. A model can clear every offline evaluation gate, beat the baseline established two posts ago in this series, and still be systematically wrong for a minority of the people it makes decisions about. Standard evaluation practice, looking at one aggregate number on a held out test set, is not built to surface this. It takes a deliberate step of breaking the evaluation down by subgroup to see it.

It also helps to look past accuracy to the error rates that make it up, since two models can share the same accuracy while distributing their mistakes very differently across groups. A useful pair of quantities are the false positive rate and false negative rate, conditioned on group membership.

$$ FPR_g = P(\hat{Y}=1 \mid Y=0, G=g) \qquad FNR_g = P(\hat{Y}=0 \mid Y=1, G=g) $$

Equalized odds is one formal criterion for group fairness: it asks that $FPR_g$ and $FNR_g$ be approximately equal across groups $g$, rather than asking that the overall accuracy be high [1].

Equalized odds is not the only formal fairness criterion in the literature, and different criteria can be mutually incompatible for a given dataset, which is itself an important and well documented result. The point to take from it for practice is narrower: overall accuracy and per group error rates are different measurements, and a governance process that only checks the first has not actually checked for fairness at all.

Why this matters even when no one intended it is worth stating plainly. A model does not need explicit malicious intent to cause real harm; it only needs to learn a pattern present in the data it was trained on, and that pattern can encode a history of unequal treatment that predates the model by decades. The absence of intent does not reduce the impact on the people affected, and it should not reduce an organization's responsibility to check for it before deployment.

Where bias enters a model

Bias rarely enters a model because someone wrote a rule to treat a group differently. It enters through the data and the process around the data, in ways that are easy to miss because they look, at every individual step, like ordinary data science.

Biased historical labels

A supervised model needs a label to learn from, and for many real world problems that label is itself the output of a past human decision process. If the historical process that generated the labels applied unequal scrutiny to different groups, the labels inherit that inequality, and the model trained on them will reproduce it faithfully, because from the model's perspective the label is the ground truth. This is not a data quality bug in the ordinary sense. The data is recorded correctly; it simply records a historically uneven process rather than the underlying reality that process was trying to measure.

Proxy features that stand in for a protected characteristic

A common and reasonable first response to fairness concerns is to exclude protected characteristics such as race or ethnicity from the data entirely. This is necessary, but it is not sufficient. Other features, individually innocuous, can correlate strongly enough with a protected characteristic that a model can effectively reconstruct the excluded signal from them in combination. A claimant's zip code, the office that handled a claim, the type of vehicle insured, or patterns in free form text can each carry a trace of a protected characteristic even though none of them is that characteristic. Removing a single column does not remove the information if that information is smeared across several other columns the model still has access to.

Feedback loops from the model's own past decisions

Once a model is deployed, its decisions can start to shape the data it is later retrained on, closing a loop that reinforces whatever pattern the model already learned. If a model routes certain claims to manual investigation more often, investigators naturally confirm more fraud where they are looking more closely, and those confirmations become next year's training labels. The pattern strengthens with every retraining cycle, regardless of whether the true underlying rate of fraud actually differs between groups, because the model's own attention, not the ground truth, is driving what gets labeled next. This is the same structural problem as the biased historical labels above, except the model itself is now the source generating the bias going forward instead of a human process that predates it.

Governance as a process

None of the three sources above shows up as a bug in code review or a failing unit test. A model with a serious fairness problem can pass every conventional software quality gate and still cause harm, which is why governance has to be a deliberate process wrapped around model development, not an assumption that careful engineering will catch it as a side effect.

Model review before deployment

A model review is a checkpoint between "the model passes offline evaluation" and "the model is live," staffed at least in part by people outside the immediate modeling team, ideally including someone who can represent the interests of the population the model will affect. Its job is to ask questions the modeling team's own success metrics do not ask by default: what does performance look like broken down by relevant subgroup, not only in aggregate; how were the training labels generated, and by what process; and which features in the dataset are plausible proxies for a protected characteristic, even an unintended one.

Documentation, sometimes called a model card

A model card is a short, structured document that accompanies a trained model and states its intended use, the population it was evaluated on, its performance broken down by relevant subgroup, and its known limitations [2].

The value of a model card is that it makes the boundaries of a model explicit and durable, rather than living only in the memory of the team that built it. A model built to flag a claim for manual review is a very different thing from a model built to automatically deny a claim, even if the underlying statistics are identical, and a model card is where that distinction gets written down so it survives staff turnover, reorganizations, and the ordinary passage of time.

A defined process to contest a decision

Even a model that has been reviewed and documented carefully will still be wrong for some individual people, because a model's fairness properties are statements about groups and averages, while any single person affected by it experiences only their own outcome. Governance therefore needs a second, complementary mechanism alongside review and documentation: a defined, accessible path for someone affected by a model's decision to contest it and have a specific human review their specific case. This is not a replacement for fixing group level bias upstream; it is a separate safety net for the individual errors that persist even in a well governed model, and its existence is itself part of what makes a model responsible to deploy.

Accountability

A governance process only functions if it terminates in a person, not a metric. Accountability means someone specific, by name or by role, is responsible for a model's real world impact on the people it touches, separately from whoever is responsible for its accuracy on a test set. An accuracy number is a property of the model. Its impact is a property of the world the model operates in, and only a person can answer for what happens in the world. Without a named owner for that impact, a review process, a model card, and an appeals path can all exist on paper while nobody is actually positioned to act when they surface a problem.

Worked example: mysurance, revisited

The first post in this series proposed a hypothesis for mysurance, "misrepresentation is different from intentional damage," and defined the two terms it discriminates between.

Misrepresentation is said to occur when a claim is made on nonexistent assets.

Intentional damage is said to occur when an insured asset is intentionally damaged.

The second post built out the measurement variables to test that hypothesis: historical variables such as cost per type of damage, textual variables drawn from a claimant's free form answers, social variables drawn from association and social interaction, and economic variables. Later posts in the series covered turning those variables into a model, scaling it from a single experiment to a general system, establishing a baseline, and monitoring it once it is live. By the time mysurance's fraud model reaches the point this post is concerned with, it has cleared every one of those steps and is, by ordinary measures, ready to ship.

Here is where the governance question has to be asked before that happens. The model was trained on years of historical claims investigation outcomes used as its label, "confirmed fraud" or not. Suppose that, for reasons unrelated to the model, historical investigators applied more scrutiny to claims from certain neighborhoods, a pattern of uneven enforcement that predates the model by years and has nothing to do with the actual rate of misrepresentation or intentional damage in those neighborhoods. Those neighborhoods will show a higher rate of "confirmed fraud" in the training labels simply because they were investigated more, not because the underlying rate of fraud was actually higher there. This is exactly the biased historical labels problem described above, and mysurance's model, trained faithfully on this data, will learn to flag claims from those neighborhoods at a higher rate even when every other detail of the claim is held constant.

The proxy problem compounds it. Suppose mysurance never includes race or ethnicity as a field anywhere in its data, by design. The historical variables from the second post in this series, cost per type of damage by category, and the social variables from that same post, association and social interaction, both correlate with neighborhood, and neighborhood correlates with the demographic composition of who lives there. The excluded characteristic is never a column in mysurance's data, but it is reconstructable from columns that are, which is why removing it as an explicit field is necessary but not sufficient. Once the model is live, the feedback loop closes the circle: claims flagged more often from these neighborhoods get investigated more often, generating more confirmed fraud labels from those same neighborhoods for the next retraining cycle, and the pattern strengthens each year whether or not the true underlying fraud rate differs at all.

What would a governance process need to look like before mysurance's fraud model could responsibly go into production? Concretely:

Model review would need to break down false positive and false negative rates by neighborhood and by other plausible proxy groupings, not just report overall accuracy against the baseline established earlier in this series, and would need to specifically test whether the historical and social variables from the measurement variables post are functioning as proxies for a protected characteristic mysurance never intentionally collected.
A model card for the fraud model would need to state its intended use plainly: the model flags a claim for manual investigation, it does not itself deny a claim, and using it to automatically deny a claim is explicitly out of scope. It would document the training data's date range and source, note the known risk that historical investigation intensity varied by neighborhood, and report evaluation results broken down by neighborhood rather than only in aggregate.
An appeals path would need to give a policyholder whose claim is flagged a clear, accessible way to request a human review of their specific case, handled by a named team, with that team's outcomes tracked over time so that a pattern of appeals succeeding disproportionately for one neighborhood becomes, itself, a signal that feeds back into the next model review cycle.
A named accountable owner, someone like a head of claims model risk rather than the data science team's aggregate accuracy metric, would need to be answerable for how the model performs across neighborhoods in production, not only for how it performed on a held out test set before launch.

None of this is a one time gate passed before launch and then forgotten. The feedback loop problem means the same review needs to repeat on a schedule for as long as the model is in production and retrained on its own downstream outcomes, using exactly the same monitoring discipline the previous post in this series described for drift and degradation, now pointed at fairness rather than only at overall accuracy.

Closing the series

Across these ten posts, one problem, mysurance's insurance fraud model, has been carried from a plain hypothesis through measurement variables, latent factors, experimental design and controlling for other factors, collection and analysis of data for pattern discovery, modeling and its numerical and qualitative refinement, scaling from a single experiment to a general system, establishing a baseline, monitoring it once live, and finally asking whether it should be trusted with real consequences at all. Each step exists because skipping it has a specific, predictable cost. Skip the hypothesis and no one agrees on what problem is actually being solved. Skip measurement variables and a model has nothing meaningful to learn from. Skip a baseline and there is nothing to know whether the model is any good. Skip monitoring and a good model quietly rots as the world it was trained on moves on without it. Skip governance and a model that is statistically accurate by every metric the team tracked can still cause real, unequal harm to real people, exactly the way mysurance's fraud model could have if it had shipped without a review, a model card, or an appeals path.

The ten steps read as a line from hypothesis to governance, but in practice they behave more like a loop than a line. A problem governance surfaces, a proxy variable, a biased label, sends a data scientist back to measurement variables and to the original hypothesis to ask whether the problem was framed correctly to begin with. That loop, returning to the start with better questions each time, is closer to what the actual practice of data science looks like than any single step in it, and it is the note this series ends on.

References

[1] Hardt, M., Price, E., Srebro, N. 2016. Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems. 29.

[2] Mitchell, M. et al. 2019. Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency. 220 to 229.

Back to posts