Lecture 1-1: The History and Working Principles of AI

3.1 Symbolic AI and the promise of the hand-coded mind

The Dartmouth researchers, and the two decades of work that followed them, pursued what is now called symbolic AI, or "good old-fashioned AI." The approach was to hand-code the rules of reasoning directly: represent facts about the world as symbols, represent logical relationships between those symbols, and let a program manipulate them according to explicit rules a human being had written down in advance. A chess-playing program built this way does not learn what a good move looks like from experience. It searches a tree of possible moves according to rules and evaluation functions its programmers supplied.

This approach produced a genuinely useful category of software in the 1970s and 1980s: expert systems, programs that encoded a narrow domain's rules — medical diagnosis from symptoms, mineral prospecting from geological data — as long chains of "if this, then that" logic supplied by human specialists. Some of these worked well enough to be used in practice. Their failure, when it came, was not a failure of ambition but of scale. Hand-coding rules for a narrow domain is tractable. Hand-coding rules for the ordinary judgment a five-year-old exercises without effort — recognizing a face at a slightly new angle, knowing that a request phrased sarcastically means something different from the same words meant sincerely — turned out to require more explicit rules than anyone could write, because most of that judgment does not run on explicit rules at all. Historians of AI call the period that followed this realization, through much of the 1980s and into the 1990s, an "AI winter": funding and confidence both receded, because the symbolic approach had run into a wall it could not hand-code its way past.

3.2 The deep-learning turn: learning the rules instead of writing them

The alternative that eventually broke through was not new in the 1980s — the mathematics of backpropagation, the algorithm that lets a multi-layered artificial neural network adjust its own internal parameters based on how wrong its outputs were, was worked out and popularized in a widely cited 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams. What backpropagation offered, in principle, was a way for a system to learn its own rules from examples rather than have them supplied by a programmer. In practice, through most of the following two decades, it was starved of the two things it needed to be useful at scale: enough labeled data to learn from, and enough computing power to do the learning in a reasonable time.

Both arrived together around 2012. That year, a neural network called AlexNet, trained on the ImageNet dataset — millions of photographs hand-labeled with what they depicted — beat every competing approach in an annual image-recognition competition by a wide enough margin that the result is now treated as the field's turning point. What changed was not the underlying mathematics of backpropagation, which had existed for decades. What changed was that a large enough labeled dataset and enough computing power, in the form of graphics processing units repurposed for this kind of arithmetic, finally let a network learn useful internal representations directly from raw data rather than from features a human engineer had pre-selected. The pattern generalized quickly beyond images: given enough data and enough computation, a network that learns its own representations tends to outperform a system built on hand-designed rules, in domain after domain.

3.3 The transformer and the architecture behind large language models

The specific architecture behind the current generation of language models arrived in a 2017 paper by Ashish Vaswani and colleagues at Google, titled, with a certain dry confidence, "Attention Is All You Need." The transformer architecture it introduced solved a practical problem earlier neural-network designs for language had struggled with: how to let a model weigh the relevance of every other word in a passage to the word it is currently processing, regardless of how far apart they sit in the text, and to do this efficiently enough to train on enormous quantities of text at once. The mechanism is called self-attention. A full technical account is not necessary for this course's purposes, but the shape of what it does is: for each word (technically, each token) the model is generating, it computes how much weight to give every other token already in the context, then produces its next output based on that weighted combination.

Trained on a large enough portion of the text available on the internet, plus books, code, and other written material, this architecture produces what is now called a large language model (LLM). It is worth being exact about what such a model is doing when it produces a sentence, because imprecision here is exactly where the theological chapters later in this course would go wrong if they inherited a mistaken picture from lecture 1. An LLM, given a sequence of tokens, computes a probability distribution over what token is statistically likely to come next, given everything it learned from its training data about which tokens tend to follow which other tokens in which contexts. It then samples from that distribution — sometimes taking the single most probable next token, sometimes a somewhat less probable one, depending on how the system is configured — and repeats the process, one token at a time, to produce what reads as a complete, often eloquent, response.

This is genuinely difficult to hold in mind while reading fluent output, because fluency is exactly the property human beings have always used as evidence of a mind behind the words. An LLM has a model of which words plausibly follow which other words. It does not have a model of which of its own sentences are true. When such a system states a fact correctly, it is because the correct fact was well-represented and consistently expressed in its training data, not because the system checked the fact against the world. When such a system states a fact confidently and incorrectly — a well-documented phenomenon usually called "hallucination," a label this course will return to in chapter 14 — it is producing exactly the same kind of output, by exactly the same mechanism, that produced its correct statements. The mechanism does not distinguish the two cases internally, because the mechanism was never built to track truth. It was built to track plausibility.

3.4 Narrow AI, and the hypothesis called AGI

Everything described so far — image recognition, language generation, game-playing systems, recommendation algorithms, medical-diagnosis-assisting software — is an instance of what the field calls narrow AI (sometimes "weak AI"): a system that performs one class of task, or a bounded set of related tasks, often at a level that meets or exceeds human performance within that narrow scope, without any capacity to transfer its competence to a task it was not built or trained for. A chess engine that plays at a level no human can beat cannot, by virtue of that ability, hold a conversation, drive a car, or recognize that the room it is being run in is on fire. Its competence is real and its scope is fixed.

Artificial general intelligence (AGI) names a different, and as of this writing entirely hypothetical, kind of system: one capable of the same kind of flexible, transferable reasoning across essentially any intellectual task that a human being can perform, learning new domains the way a person does rather than requiring a separate training process built for each one. It is essential for this course, and for careful thinking about AI generally, to keep these two categories from blurring into each other. Narrow AI exists, is powerful, and is already reshaping the tasks this course will examine in later chapters — bias in decision systems, labor displacement, medical diagnosis, sermon preparation. AGI does not yet exist. Every claim in the chapters that follow about narrow AI's actual behavior rests on systems that can be inspected, tested, and audited today. Claims about AGI, by contrast, are claims about a system nobody has yet built, and their theological and practical weight has to be assessed accordingly — carefully, and separately from claims about what current systems do.

3.5 A skeptic's corrective: Gary Marcus and the limits of scale

It would be a mistake to end this lecture as though the field agreed that the path from narrow AI to AGI is simply a matter of adding more data and more computing power to the current architecture. It does not agree on this, and the disagreement is not confined to theologians or outside critics — it runs through AI researchers themselves. Cognitive scientist Gary Marcus has argued consistently, across a series of books and public writing since the 2010s, that deep-learning systems, transformers included, exhibit a specific and recurring weakness: they are excellent at interpolating within the distribution of patterns they were trained on and poor at the kind of systematic, compositional reasoning that lets a human being reliably apply a rule to a genuinely novel situation. Marcus's point is not that current systems are unimpressive — he does not dispute that they are — but that impressive performance on the tasks a system was effectively trained to be impressive at is weak evidence for the claim that scaling the same architecture further will eventually produce general intelligence. On his account, something structurally different, not merely larger, is probably required, and treating "more scale" as a settled roadmap to AGI mistakes an engineering achievement for a solved scientific problem.

This course takes no side in that dispute as a matter of AI research — it is a technical question this course is not positioned to settle, and lecture 1-2 will show that serious, informed people disagree sharply about it. What this lecture insists on is narrower: a Christian engaging AI seriously should know that the confident predictions circulating about AGI's arrival are contested by people whose job is building the systems in question, not only by outsiders uneasy about the technology on other grounds.

3.6 Sound counsel and considered ways

Proverbs 1:5 commends the "man of understanding" who listens in order to "attain to sound counsel" — a posture of taking in information before forming a judgment, which is exactly what sections 3.1 through 3.5 of this lecture have tried to model with respect to a technology that generates strong opinions faster than it generates careful ones. Proverbs 14:15 sharpens the point with a contrast: "a simple man believes everything, but the prudent man carefully considers his ways." Applied to this lecture's subject matter, the contrast is not between believing AI enthusiasts and believing AI skeptics — it is between believing either camp's confident claims without first understanding, at the level this lecture has tried to supply, what the technology in question is actually doing. A reader who finishes this lecture unable to state, in one sentence, why "an LLM predicts the next token" is a substantive claim and not a throwaway technical detail has not yet met the standard Proverbs 14:15 sets, and the rest of this course will be harder to engage well as a result.

Summary table