Lecture 6-1: The Structure of Algorithmic Bias

Leviticus 19:15 — "You shall do no injustice in judgment. You shall not be partial to the poor, nor show favoritism to the great; but you shall judge your neighbor in righteousness." (World English Bible)

Deuteronomy 25:13 — "You shall not have in your bag diverse weights, one heavy and one light." (WEB)

3.1 Three pathways, not one

"Algorithmic bias" is sometimes discussed as though it were a single phenomenon with a single fix. It is not. At least three distinct mechanisms are worth separating, because each calls for a different kind of scrutiny and a different kind of remedy.

Biased training data. A machine learning model does not reason toward a conclusion the way a person weighing evidence does; it finds statistical patterns in a large set of past examples and extrapolates them forward. If those past examples were themselves produced under discriminatory conditions — a hiring dataset built from a company's actual past hires, when that company's actual past hiring favored one group over an equally qualified other; a policing dataset built from arrest records, when those arrests reflect decades of over-policing of particular neighborhoods — the model does not correct the discrimination. It learns it as a pattern and reproduces it with the appearance of statistical objectivity. This is the most widely understood pathway, and also the easiest to state too simply: it is not that "bad data" is a rare contamination of an otherwise neutral process. Historical inequality is present, to some degree, in nearly every large dataset drawn from real human institutions, because those institutions operated under conditions of unequal treatment for most of their history. The question is never whether historical bias is present in training data; it is how much, and whether anyone checked.

Proxy variables. Even when a system is explicitly barred from using a protected category directly — many jurisdictions prohibit lenders from using race as an input, for instance — a model trained on enough other variables can reconstruct the excluded category through correlation. Zip code is the standard example: in a country with a long history of racially segregated housing, zip code correlates strongly with race, so a model that uses zip code as an input can produce racially disparate outcomes while never once being told anyone's race. The same has been shown for variables like name (correlated with ethnicity), certain purchase histories, and even typing patterns on a mobile device. A system can be, in the most literal sense, "colorblind" in its inputs and racially disparate in its outputs, and both statements can be true at once. This is the technical heart of why "the algorithm doesn't know anyone's race" is not the defense it sounds like: not knowing a category directly is compatible with reconstructing it indirectly, at scale, automatically, without any human involved ever deciding to do so.

Feedback loops. The third pathway is the one that turns a one-time bias into a self-reinforcing system. Suppose a predictive-policing algorithm flags a neighborhood as high-risk based on past arrest data, and police departments respond by patrolling that neighborhood more heavily. Heavier patrolling produces more arrests in that neighborhood — not necessarily because more crime is occurring there, but because more police are looking. Those new arrests become new training data, which confirms and often sharpens the original prediction, which justifies still heavier patrolling. The loop does not require anyone in it to hold a discriminatory intention at any point; the system trains on its own output and each cycle can compound the disparity it started with. Kristian Lum and William Isaac's 2016 study of predictive policing in Oakland, California, documented exactly this dynamic: the algorithm they tested would have directed policing resources overwhelmingly toward low-income, predominantly Black and Latino neighborhoods, at a rate far exceeding the actual estimated drug use in those neighborhoods compared to more heavily white areas of the city. A feedback loop is bias with a memory — the more a biased system is used, the more confidently biased its next output becomes.

3.2 A documented case: Gender Shades

Abstract description invites the objection that this is all speculative or exaggerated. It is not, and the Gender Shades study is a good instance of why.

In 2018, Joy Buolamwini (MIT Media Lab) and Timnit Gebru (then at Microsoft Research) published Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, testing three commercial facial-analysis systems — from IBM, Microsoft, and the Chinese company Face++ — on their ability to correctly classify a subject's gender from a photograph. The study's methodological innovation was to test accuracy not just by gender or by skin tone separately, but by the intersection of both, using a specially built dataset balanced across four categories: lighter-skinned men, lighter-skinned women, darker-skinned men, and darker-skinned women.

The results were stark. All three systems performed best on lighter-skinned men, with error rates under one percent. All three performed markedly worse on darker-skinned women — one system's error rate for that group exceeded thirty-four percent. The gap between the best-served and worst-served group, on some systems, was greater than forty percentage points. This was not a matter of the technology being generally imperfect; it was a matter of the technology being reliably accurate for one demographic and unreliable for another, in the same product, sold as a single tool.

The likely cause traces back to the first pathway above: the benchmark datasets historically used to train and evaluate facial-analysis systems were themselves disproportionately composed of lighter-skinned, male faces, so the systems had, in effect, been optimized against a training and testing standard that underrepresented the group they performed worst on. No engineer needed to hold any animus toward darker-skinned women for this outcome to occur; an unrepresentative dataset, unaudited, was sufficient. The study's practical effect was significant: the companies involved (IBM and Microsoft notably) revised their systems and testing practices, and Gender Shades is now a standard citation across the algorithmic-fairness literature for the claim that accuracy is not evenly distributed by default — it has to be checked for, because it will not check itself.

3.3 "Just doing math" is not a defense — it is the problem restated

The claim this section has been building toward is simple to state and easy to underestimate: a system that is mathematically consistent can still be unjust, because consistency describes how reliably the system applies its rule, not whether the rule itself is fair. A facial-recognition system with a thirty-four-percent error rate for one group and near-zero for another is being perfectly consistent — consistently worse for one group, every time. Mathematics does not launder that outcome into neutrality; it just makes the outcome harder to see, because a percentage looks more objective than a person's judgment does, even when the percentage was produced by the very kind of historically conditioned pattern that would be recognized instantly as unfair coming from a human decision-maker.

This is precisely the false comfort Scripture's justice legislation was written against — not against mathematics, which did not exist in the relevant sense, but against the specific temptation to trust an instrument's appearance of objectivity over an honest examination of what it actually produces.

3.4 Honest weights, then and now

Leviticus 19:15 sits inside the Holiness Code's instructions for how Israel's own courts were to function: "You shall do no injustice in judgment. You shall not be partial to the poor, nor show favoritism to the great; but you shall judge your neighbor in righteousness" (WEB). The verse is notable for refusing partiality in either direction — not only the more obvious favoring of the powerful, but also a sentimental favoring of the poor that would equally corrupt the verdict. Justice, in this text, is not measured by which side benefits; it is measured by whether the judgment tracks the actual facts of the case, applied the same way to everyone brought before it.

Deuteronomy 25:13-15 makes the same demand in a commercial rather than judicial setting: "You shall not have in your bag diverse weights, one heavy and one light... you shall have a perfect and just weight." A merchant who kept two sets of weights — a heavy one for buying, so he received more than he paid for, and a light one for selling, so he gave less than he charged for — could present each individual transaction as a simple, consistent application of "the weight." Every single sale would look mathematically unremarkable to the customer standing at the scale. The dishonesty was not in any one transaction; it was built into the instrument itself, invisible unless someone thought to check the weights against a known standard.

That is a strikingly exact description of what a proxy variable or an unaudited training set does inside a modern algorithm. No one transaction — no single loan denial, no single facial-recognition failure — looks like fraud from the inside. Each output can be, and often is, defended as "just what the model calculated." The Deuteronomic instruction does not accept that defense from the merchant, and this lecture's argument is that it should not be extended to the algorithm either. The remedy the text prescribes is not a general appeal to good intentions; it is a concrete, checkable practice — "a perfect and just weight" — which is to say, an audited one. An algorithm is, in this sense, exactly a set of weights in a bag, and the ancient demand was never really about weights. It was about refusing to let an instrument's appearance of neutrality substitute for someone actually checking it.