Lecture 3-1: Reason versus Computation

Proverbs 1:7 (World English Bible): "The fear of Yahweh is the beginning of knowledge; but the foolish despise wisdom and instruction."

1 Corinthians 8:1-2 (WEB): "Now concerning things sacrificed to idols: We know that we all have knowledge. Knowledge puffs up, but love builds up. But if anyone thinks that he knows anything, he doesn't yet know as he ought to know."

3.1 What an LLM is actually doing

It is worth restating precisely, because imprecision here is where most bad AI theology starts. A large language model is trained by being shown enormous quantities of text and adjusting billions of internal numerical parameters so that, given the text so far, it assigns a higher probability to the token that actually came next in the training data. Once trained, it generates new text by repeating a simple operation: given everything generated so far, compute a probability distribution over the next possible token, and sample from it. There is no internal representation labeled "this claim is true" that the system consults before it speaks. There is only "this continuation is statistically well-supported by patterns in the training data, given everything said so far." A model can therefore produce an extremely well-formed, confident-sounding argument for a false claim with exactly the same internal process it uses to produce a well-formed argument for a true one, because "well-formed and confident-sounding" is the target it was actually trained toward, and "true" is not directly its target at all.

This is not a criticism of the engineering — it is a description of what the engineering is for. The question this lecture asks is what follows, philosophically, from that description.

3.2 The Chinese Room, developed further

Lecture 1-3 introduced John Searle's Chinese Room argument (1980) as a standing objection to strong claims about artificial intelligence. It is worth developing more fully here, because the argument was built for exactly this question.

Searle asks you to imagine a person who does not understand Chinese, sitting in a room. Chinese characters come in through a slot. The person has a large rulebook, written in English, that says things like "when you see this sequence of characters, write down that sequence of characters and pass it back out." Following the rulebook mechanically, the person can produce responses so appropriate that a Chinese speaker outside the room is convinced they are corresponding with someone who understands Chinese fluently. But the person in the room understands no Chinese at all. They are manipulating uninterpreted symbols according to formal rules — matching shapes to shapes — and the meaning of those shapes never enters the process anywhere.

Searle's target is the claim, associated with what he called "strong AI," that a computer running the right program does not merely simulate a mind but has one — that correctly manipulating the symbols is sufficient for understanding. His argument is that the room as a whole (person plus rulebook plus slot) is functionally equivalent to a computer running a natural-language program, and yet it manifestly does not understand Chinese, however convincing its output. If the room does not understand, then a computer that works the same way — matching input patterns to output patterns according to rules built into its architecture — does not understand either, no matter how good the output looks from outside.

Applied to a large language model: predicting the next token that best continues a pattern in the training data is a much more powerful and much more statistically sophisticated version of following a rulebook, but it is a version of the same thing Searle describes — matching patterns to patterns, where the patterns are made of words and the words are not understood as referring to anything by the system doing the matching. The model manipulates tokens that are correlated with meaning, for us, without those tokens meaning anything to it.

It is important to be precise about what this argument does and does not establish. It does not prove that no physical system other than a biological brain could ever understand anything — that is a much larger claim, and Searle himself did not think biology was magic, only that whatever produces understanding, mere formal symbol manipulation by itself is not sufficient for it. What the Chinese Room does establish is a burden of proof: producing the right outputs is not, by itself, evidence of understanding, and anyone claiming an AI system understands language needs an argument beyond "look how good its answers are."

3.3 The "stochastic parrots" critique and the functionalist reply

The Chinese Room is a thought experiment from 1980, aimed at a hypothetical program. Emily Bender and her co-authors (in the widely cited 2021 paper written with Timnit Gebru and others, commonly known by its central phrase, "stochastic parrots") made a version of the same point aimed directly at large language models as they actually exist. Their argument is that a language model is, in a precise sense, a system for haphazardly stitching together sequences of linguistic forms it has observed in its training data, according to probabilistic information about how they combine, without any reference to meaning: a stochastic parrot repeats plausible-sounding sequences without understanding what they refer to. Bender's related "octopus test" thought experiment (developed with Alexander Koller) makes the same point in a different image: an octopus that has only ever eavesdropped on two people's cable conversations, with no access to the world those people are talking about, could in principle learn to produce statistically plausible responses to new messages — without ever understanding a single word, because it has never had any contact with what the words are about. Meaning, on this view, requires a connection to a world beyond the text, which a system trained only on text does not have.

The functionalist reply, made by many researchers in machine learning and cognitive science, takes the strongest form of this objection seriously rather than dismissing it, but argues the underlying picture of "understanding" it assumes is not obviously correct. Functionalism, discussed already in 1-3, holds that mental states are defined by the functional role they play — what causes them, what they cause, how they relate to other states — rather than by what they are made of. A committed functionalist can grant that a language model's training is purely textual and still argue that if a system's internal states come to play the right functional role (tracking claims, updating on contrary evidence, supporting appropriate inference), that is what understanding consists in, and demanding some further ingredient — a connection to "the world," a biological substrate, a felt quality of insight — is asking for something that, on functionalist grounds, was never a coherent requirement to begin with. This is a genuine, unresolved argument, and this course does not pretend the functionalist position has no serious defenders; it is one of the main live positions in contemporary philosophy of mind. This lecture's own weight falls, with Searle and Bender, on the view that grounding in a world and genuine intentionality are not optional extras but the very thing "understanding" was supposed to name — but a fair reader should recognize that this is a chosen side in an active argument, not a settled fact reported from outside it.

3.4 Biblical knowing is not merely informational

Turn now to the two texts for this lecture, and notice something they share: neither treats "knowledge" as a purely informational category, settled once the right facts are stored.

Proverbs 1:7 states that "the fear of Yahweh is the beginning of knowledge." This is a claim about where knowledge starts, and it locates the starting point in a posture — reverence toward God — not in an accumulation of data. On this account, someone could hold an enormous quantity of correct propositions and still not have "knowledge" in the sense Proverbs means, if the reverence that is supposed to be its foundation is missing. This is not a claim that information does not matter; it is a claim that information is not the whole of what "knowing" names in the wisdom literature.

1 Corinthians 8:1-2 goes further and makes the same point about a specific, contested question in the Corinthian church (whether it was permissible to eat meat that had been sacrificed to idols). Paul concedes the point his readers are actually right about — "we all have knowledge" — and then immediately relativizes it: "knowledge puffs up, but love builds up. But if anyone thinks that he knows anything, he doesn't yet know as he ought to know." The passage does not deny that the Corinthians know true things about idols (elsewhere Paul agrees with their theology on this exact question). It denies that correct propositional content, by itself, constitutes knowing "as he ought to know." Right information held without love is, on Paul's own account, a kind of knowing that falls short of what knowing is for.

Put the two texts together and a pattern emerges that is directly relevant to this lecture's argument: the Bible's own vocabulary for "knowing" already refuses to treat it as a purely informational state, long before anyone had a reason to ask the question about a machine. A large language model, whatever else it does, produces text that tracks statistical patterns in a training corpus. Even in the most generous case — granting, for the sake of argument, that its outputs somehow constitute a functional analogue of belief — it has no candidate for reverence toward God, and no candidate for love of a neighbor. It is not merely that the model lacks understanding in Searle's sense; on the Bible's own terms, correct information without either of those postures was never going to count as "knowing as one ought to know," even for a human being. The theological point does not settle the philosophy-of-mind debate from section 3.3. It does something different and, for a Christian, more important: it insists that even a system that fully won that debate — even a system that somehow did understand its outputs — would still not thereby possess the fuller thing Scripture calls knowledge.