Day-to-day, if you engage with AI, or talk about it, you’re encouraged to treat it as some sort of person. Its design impels you to text or talk to it as you would to a human. When it shows it’s “thinking,” it says “I need to define the scope of the topic, and when it answers, it says “I found X” and “I apologize for the error.”
All of this is easy to accept. After all, for our entire lives, putting together an elegant answer to our questions was a sure sign that the putter-together was (a) intelligent and (b) trying to tell us something.
Leif Weatherby thinks this metaphor is a colossal mistake. In reality, he argues in his recent book, Language Machines, Large Language Models like Claude and ChatGPT, which have no minds or intentions, prove that language requires neither. So there is no point in taking either side of a debate about whether AI is human-like. (No need to argue about how smart AIs are, or how ethical, or how intent they are on escaping human oversight, or whether they’re going to kill us or provide us with Utopia, or whether they have rights). We can accept that AI produces meaningful language without thinking it must be on its way to consciousness; we can accept that it’s just a machine without insisting that its product must be worthless.
Instead, Weatherby says, we should be looking at what these “culture machines” tell us about the culture and politics that shape us. Instead of ethicists and philosophers of mind, he looks to literary theorists (on the nature of language) and political and economics scholars (on the way AI is used to define and control markets and keep labor in line).
I wanted to speak with him because it seems to me the AI-as-person metaphor has taken over most news and conversation about this technology. Since AI is not really a person, we need other ways to think about it, and talk about it. And Weatherby’s take is rigorous, deeply informed and, for me, mind-expanding.
Some Useful Background
Here’s some background about some of the things that came up in the conversation.
“Stochastic parrots” - This is shorthand for the claim that a Large Language Model is “a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning”. Introduced in a 2021 paper by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell. As Bender wrote this year, the claim here is that this isn’t real language because there is no intention to communicate in the model’s mere “remix” of training data, its instructions, and the user prompts it received. (Weatherby, who doesn’t believe that meaning depends on some human’s intentions, is not a fan.)
“InstructGPT” - A term for the OpenAI GPT-3 models that were the basis for ChatGPT when it was released in November of 2022.
Claude’s constitution - A “detailed description of Anthropic’s intentions for Claude’s values and behavior,” which is part of its training.
Mid-training - the phase in the development of a large language model in which its trainers focus on specific skills and practices they want it to have. This occurs after “pre-training,” when the model goes to work on vast amounts of text, to find statistical patterns in all language use. Claude’s constitution is used in mid-training.
Context window - The maximum amount of text that the model can consider at one time when generating output. Advanced models now can hold hundreds of thousands of words in a context window (for example, all 25,000 words of Claude’s constitution). Human brains don’t work the way LLMs do, but it’s safe to say humans have smaller context windows than models.
Reminder: The transcript is AI-generated and likely contains some mistakes. If you spot a bad one, please let me know.
