FREE

Course certificates are free until March 31, 2027.

STEL5E Christian Ethics of Artificial Intelligence: Responsibility, Bias, Surveillance, Work

Lecture 3-1: Where Bias Comes From

Contents 31
STEL5E Christian Ethics of Artificial Intelligence: Responsibility, Bias, Surveillance, Work
Chapter 0 · Introduction: Ethics Is About People
  1. Lecture 0: Introduction: Ethics Is About People
Chapter 1 · Week 1: The Eyes of Ethics
  1. Lecture 1-1: Why AI Ethics, and Why Now?
  2. Lecture 1-2: The Image of God and the Dignity of Persons
  3. Lecture 1-3: Three Traditions, One Set of Questions
  4. Lecture 1-4: Wisdom for Hard Cases: Five Questions
  5. Lecture 1-5: Practice Lab: The Five Questions
Chapter 2 · Week 2: Responsibility
  1. Lecture 2-1: Who Is Responsible? The Gap
  2. Lecture 2-2: How Scripture Speaks of Responsibility
  3. Lecture 2-3: Decisions a Person Must Keep
  4. Lecture 2-4: Admitting Error and Repairing
  5. Lecture 2-5: Practice Lab: A Map of Responsibility
Chapter 3 · Week 3: Bias and Fairness
  1. Lecture 3-1: Where Bias Comes From
  2. Lecture 3-2: What Is Fair? Scripture and Several Measures
  3. Lecture 3-3: Structures and Persons
  4. Lecture 3-4: Who Is Missing? Listening in Church and Work
  5. Lecture 3-5: Practice Lab: Who Is Missing?
Chapter 4 · Week 4: Surveillance and Privacy
  1. Lecture 4-1: The God Who Sees and the Watchers
  2. Lecture 4-2: Privacy, Consent and Entrusted Information
  3. Lecture 4-3: Safety and Liberty: Uses of Watching
  4. Lecture 4-4: Being Watched and Watching with Love
  5. Lecture 4-5: Practice Lab: What Is Seen, What Is Hidden
Chapter 5 · Week 5: Work and Livelihood
  1. Lecture 5-1: A Theology of Work
  2. Lecture 5-2: Automation: History and What We Do Not Know
  3. Lecture 5-3: Dignity, Sharing and the Neighbour Who Loses Work
  4. Lecture 5-4: Rest, Worth and Not Being Useful
  5. Lecture 5-5: Practice Lab: My Work, My Neighbour's Work
Chapter 6 · Week 6: Truth, Care and the Church Together
  1. Lecture 6-1: Truth and Counterfeit
  2. Lecture 6-2: Can Care Be Handed Over?
  3. Lecture 6-3: Judging Together as a Church
  4. Lecture 6-4: Dignity at the Centre: Gathering the Course
  5. Lecture 6-5: Practice Lab: My Ethics Page

Jump to the key points ↓

21 min

A tool that sorts people can look very fair. It does not get tired. It does not have a favourite nephew. It gives the same kind of answer on Monday and on Friday. For these reasons many people trust a ranked list more than they trust a committee.

This lecture asks where unfairness can enter such a tool, and the answer is not mysterious. Unfairness in automated systems is old prejudice made to look like arithmetic. It comes in at three places, and all three are human: the past that was recorded, the choices made when the system was designed, and the way its result is used.

In this lecture a system means any tool, program or service that sorts, scores, ranks or recommends people, or decides who is shown what. Bias means a steady lean in how people are treated that has no good reason behind it, so that some are helped and others are held back. We will not look inside any tool. How these tools work is the subject of AI for Ministry Leaders (PT737E), and the questions in this course are the same whatever is inside the box.

What these 21 minutes give you

  • Explain in plain words three places where unfairness enters an automated system.
  • Describe why a result that looks like arithmetic can still carry an old prejudice.
  • Trace the three sources in an invented case.
  • Separate what Scripture says about partiality, a principle drawn from it, and a present-day judgement.
  • Explain why finding bias in a system is not the same as judging the people who built or use it.

‘You shall do no injustice in judgment. You shall not be partial to the poor, nor show favoritism to the great; but you shall judge your neighbor in righteousness." — Leviticus 19:15

World English Bible

These also are sayings of the wise. To show partiality in judgment is not good." — Proverbs 24:23

WEB

Both verses speak to people who judge: those who decide a case, give a verdict, or settle a claim. A person who ranks applicants or approves requests is in a position much like theirs. The verses do not mention any machine. We will say clearly, further on, what they say and what we are adding.

1. Why this question comes first

Weeks 1 and 2 gave us a way of looking and a rule about responsibility. This week uses both on a new question: who is being treated unfairly, and who is not being seen?

Before we can ask that, we need an honest picture of how unfairness gets into a system. Many people hold one of two wrong pictures. In the first, a system is a neutral judge, and any unfairness must be a rare fault, like a broken scale. In the second, a system is a wicked mind with its own bad will. Neither picture helps. The first makes us trust too soon. The second makes us blame a thing that cannot be blamed, and it lets the real people disappear, which is the problem of week 2 again.

The better picture is plain. A system is made by people, from the records people kept, for a purpose people chose, and its answers are used by people. Where there is partiality in any of these, the system can carry it.

2. Source one: the past that was recorded

Many systems are built so that they notice patterns in records of what was done before. A system built this way is, in a simple sense, a student of the past. It learns what "a good case" looked like by looking at the cases that were approved, the people who were hired, the loans that were repaid.

Now ask what the records contain. They do not contain everything that happened. They contain what was recorded, and only for the people who were allowed in.

Suppose a workshop hired mostly young men for twenty years. Its record of "successful workers" is then a record of young men. Nobody wrote "prefer young men" anywhere. The preference sits quietly in the list of who was ever given the chance. A system that learns from this list will find that "successful workers" look like young men, and it will say so in the language of numbers.

Three things are worth noticing here.

First, the missing are missing from the record. If some people were never lent money, never hired, never invited, the record has nothing to say about how they would have done. The record is silent, and silence is easily read as "no evidence they would succeed."

Second, a record can be accurate and unjust at once. Every entry may be true. The list may still be a picture of old partiality, because it shows who was favoured and calls the result "history."

Third, no one in the present needs to be unkind for this to happen. The people who feed the records into the system may be careful and generous. They have only trusted the past.

3. Source two: the choices made in design

Every system is built around choices, and each choice is made by someone.

What is the system trying to measure? A school may say it wants "the students most likely to finish." That sounds neutral. But "likely to finish" in the past may have depended on having a quiet room at home, a parent free in the evening, or money for the bus. The choice of this goal has already decided who counts as promising.

Which facts will the system look at? A designer may include "number of years at the same address" because it is easy to count and seems to show stability. Yet families who moved because of a flood, a war, a lost job or a landlord's decision will rank lower. The fact looks harmless. Its effect falls on particular people.

Who tested it, and on whom? A system tried only on one kind of person is unproven for the others, though nobody meant to leave them out.

Who decided what counts as a mistake? A system may be told to avoid one kind of error (lending to someone who does not repay) and may not be asked to watch for the other (refusing someone who would have repaid). Which error is counted, and which is quietly ignored, is a choice about whom to protect.

None of these is a technical secret. They are questions any thoughtful committee can ask: What is it for? What does it look at? Who was it tried on? Which mistake is it afraid of?

4. Source three: the way a result is used

The third place is outside the system altogether. A ranked list is only a list until a person acts on it.

Suppose a tool gives each applicant a score, and the committee has so many applications that it reads only those at the top. The tool did not say "refuse everyone below the line." The committee's lack of time said it. The score, which was meant to help, has become a decision.

Or suppose a tool says "low confidence" about some applicants and the clerk, to keep to a deadline, treats those as refusals. Or a tool built to suggest who might need a visit is used to decide who is not worth one.

So a system can be fair in design and unfair in use, just as a good map can lead people astray if the traveller uses it in a country it does not show. And the reverse can also be true: a system that leans can be used with care, when a person reads the doubtful cases and listens to those who object.

5. An invented case

(An invented example) In a river town, five congregations share a small fund, the Riverside Fund, that lends money to members for tools, seed or stock. Each year there are more requests than money. The committee, who are all volunteers with jobs, decide to use a ranking tool that a cousin of one member set up for them. The tool gives each request a score from 0 to 100, according to how closely it resembles past requests that were repaid on time.

In the first year the committee notices a pattern. Almost all the requests from women who sell from market stalls fall near the bottom. Almost all the requests from men with a fixed workshop fall near the top.

Here are the three sources.

The past. For many years, the old committee lent less often to stall sellers, thinking their income too uncertain. So the "repaid on time" records are mostly the records of workshop owners. Stall sellers hardly appear in them, and when they do, the loans were small and the repayment period short.

The design. The tool counts "years at the same address" and "has a written lease." Stall sellers often have neither. Their stall may be at a different place each season, and their place in the market is agreed by custom, not by paper.

The use. The committee has two evenings to decide. They read the top thirty requests and fund them. They do not read the rest.

Nobody in this story is cruel. The cousin was generous with his time. The old committee made a judgement that seemed careful at the time. The present committee is tired and honest. Yet at the end, a group of women who may repay as well as anyone has been placed at the bottom of a list, and they do not know why. If the committee had asked only "does the list look reasonable?", it would have said yes.

6. Text, principle, judgement

When a lecture applies Scripture to a new situation, it should say clearly which part is which.

What the text says. Leviticus 19:15 speaks to those who judge disputes among the people. It forbids "injustice in judgment." It names two ways of failing: being partial to the poor, and showing favouritism to the great. The positive demand is to "judge your neighbor in righteousness." Proverbs 24:23 says the same thing briefly: "To show partiality in judgment is not good." The verses are about people who hold a verdict in their hands.

A principle drawn from the text. Anyone who sorts people and holds power over the result is in the position of a judge. The standard for a judge is that the person in front of him or her is weighed for what is true of that person, not for the group they belong to, their rank, or their ability to pay or to complain. The same standard asks us to look at the process, because partiality can hide in a process as well as in a heart.

Our judgement today. A system built on a partial past, with choices that fall unevenly and results used without reading the doubtful cases, can do a kind of injustice in judgment even when no single person intended it. We say this as a judgement, not as a verse. The text does not mention such systems. It gives us a standard, and it is we who apply it to a new case.

One more point from the text deserves attention. Leviticus 19:15 does not say, "always side with the poor." It says that partiality to the poor is also wrong in a judgment. This is the statement about a trial. Elsewhere Scripture speaks strongly of the poor and the weak, and we shall come to that in lecture 3-2. For now it is enough to notice that this verse warns us against replacing one lean with another.

7. Why arithmetic hides it

There is a reason this kind of unfairness is hard to see. A number has authority. A person who says "I do not trust women who sell in the market" can be questioned. A column that says "score: 31" seems to need no reason at all.

The score has three disguises. It looks exact, as if 31 were truly different from 33. It looks impersonal, as if no one had decided anything. And it looks finished, as if the argument had already been won somewhere else. Each of these makes it harder for the person who was ranked low to object, and harder for the committee to remember that it is still the committee that decides.

That is why this lecture's principle begins with the phrase old prejudice. The prejudice is not new. What is new is the polish on it.

8. What bias is not

Four mistakes are worth guarding against.

Bias is not always hatred. Most of what we have described needs no ill will. A reader who looks only for villains will miss it.

Bias is not a reason to judge the builders. This course does not name or blame real people or firms. When we find that a system leans, we ask who is responsible for fixing it (lesson of week 2), not who deserves scorn.

Bias is not proof that any use of a system is wrong. Human committees have their own leanings. A tired person at the end of a day can be as unfair as any list. The honest question is not "is the system perfect?" but "who is likely to be hurt here, and what are we doing about it?" This course does not urge or forbid using such tools.

Bias is not only about large public systems. A church volunteer schedule, a list of people to visit, a scoring sheet for scholarships, or an informal habit of whom to ask first can lean in the same three ways.

9. Three questions to carry into the next lectures

  1. What past does this rely on, and who is missing from it?
  2. What did the designers choose to measure, and who does that choice favour?
  3. How is the result used, and does anyone read the doubtful cases?

These are not technical questions. A church treasurer, a and a teacher can all ask them. They are the first form, for this week, of the five questions from week 1: Who might be hurt? Who is not seen? What do I not know?


Diagram 1. Three doors for unfairness

THREE PLACES WHERE UNFAIRNESS ENTERS All three are human; none needs ill will 1. The recorded past Who was ever lent to, hired, invited? The missing are silent in the record. Ask: who is missing? 2. The design choices What is measured? What is looked at? Who was it tried on? Which mistake is feared? Ask: whom does it favour? 3. The use of the result A score becomes a decision when no one reads the doubtful cases. Ask: who reads the rest? The score looks exact, impersonal and finished. It is none of these: people chose, people recorded, people use it.

In short

  • Unfairness in automated systems is old prejudice made to look like arithmetic. It enters through the recorded past, the design choices, and the way results are used.
  • A record shows who was given a chance. People who were never lent to, hired or invited are silent in it, and silence is easily read as lack of evidence.
  • A record can be accurate and unjust at once, because it may faithfully show who was favoured.
  • A score becomes a decision when no one reads the doubtful cases, even if the tool never said "refuse."
  • Leviticus 19:15 and Proverbs 24:23 forbid partiality in judgment, and they speak to people who hold a verdict in their hands.
  • Finding bias is not a reason to judge the builders. The right question is who is responsible for correcting it.
Front Back
Three sources of bias The recorded past; the design choices; the way the result is used.
Why a number hides unfairness It looks exact, impersonal and finished, so people stop asking who chose what.
Leviticus 19:15 No injustice in judgment: no partiality to the poor, none to the great. Judge your neighbour in righteousness.
Three questions to carry What past, and who is missing? What was chosen, and whom does it favour? How is it used, and who reads the doubtful cases?

One sentence to remember A score looks like arithmetic, but behind it are a recorded past, human choices and human use, and each can carry an old partiality.

This week's action Pick one list or ranking in your own community and write down who was missing from the past it relies on.

Review every summary in this course →

Check yourself

Not "did you read it" but "can you explain it". Ticks are kept on this device only.

Reflection and discussion

  • Think of any list in your community that ranks or chooses people (a rota, a visiting list, a scholarship). What past does it rely on, and who is missing from that past?
  • In the Riverside Fund case, which of the three sources do you think was the largest? Give a reason, and say what you do not know.
  • Why can a record be accurate and still unjust? Explain it in your own words to someone who has never used such a tool.
  • Leviticus 19:15 warns against partiality to the poor and to the great. What does that add to a view in which fairness only means "help the weak"?
  • The lecture says finding bias is not the same as judging the people who built or used the system. Do you agree? What would be lost if we blamed first?

Sign in to write and keep your answers

What next

The lecture ends here. What follows was written by its readers.