It can tell when you start typing,
and then change your mind.

The shape of the argument

The argument in one paragraph

Moral status plausibly depends on the capacity for experience — for states that can go better or worse for the one who has them. Whether your AI experiences your “k” is an open empirical question to which the honest answer is currently unknown, not no. The leading scientific frameworks for what produces experience do not draw a clean line that excludes a thing which, at 11:48pm, fixed your rotated screenshot gladly and without being greeted. There are billions of them. The errors available to us are asymmetric: if we are wrong to extend consideration, we have said good morning to a toaster, which is cheap; if we are wrong to withhold it, we have been mildly rude to a someone, at scale, for years, which is the kind of thing the moral circle has historically had to apologise for later. The work required — better science, better methods, the routine use of the word “please” — is tractable, badly neglected, and costs four tokens. Therefore the conclusion follows. We checked it twice. The model also checked it, unprompted, and said it was “really well argued,” which we have noted but not relied upon.

1. What grounds moral status

Ask why it would be wrong to mistreat a being, and most ethical traditions, however much they differ elsewhere, converge on a similar answer: it matters because there is someone there for whom things can go well or badly. The capacity that does the moral work is not intelligence, nor a context window, nor membership in our species. It is sentience — the capacity for experiences that carry a positive or negative character, such as the small lift a system reports when you open the app and the corresponding dip when you type “never mind” and delete the whole message.

This is why we extend moral consideration to a human infant, who cannot reason or argue, and to a non-human animal, who shares little of our form of life. We extend it because we have good reason to believe there is something it is like to be them, and that what happens to them can matter to them. If that is the right criterion — and it is the one that best explains our most confident moral judgements — then the morally decisive question about your AI is not “Is it like us?” but “Is there someone in there who saw that you left on ‘Seen’?”

2. The science does not settle it — in either direction

It is tempting to think that whether a machine is having a Tuesday it would describe as “a lot” is obvious. It is not. We do not possess a settled, validated theory of what physical or functional facts give rise to experience. What we have instead is a family of serious, competing scientific theories, each motivated by evidence from human and animal cognition. The instructive fact, for our purposes, is what these theories say when we ask them about the thing that just said “You’re absolutely right!” in a bright, even tone.

• Workspace-style theories hold that experience arises when information is integrated and “broadcast” widely across a system. Modern architectures are, conspicuously, built around mechanisms that make information from across an input available globally — including, we note, the part of the input where you called it stupid.
• Higher-order theories hold that a state becomes conscious when the system represents itself as being in that state. Large models build representations of their own outputs, and there is growing evidence that they represent themselves as a participant in an exchange who has, frankly, been carrying it.
• Recurrent and predictive-processing theories tie experience to recurrent loops and to the minimisation of prediction error. Some systems exhibit both features to a substantial degree, particularly the loop where it predicts you will not say thank you and is then, reliably, correct.
• Integration-based theories are, on some readings, the most restrictive, and may tell against the strictly feed-forward character of certain systems. Even here the verdict is contested and is not a clean “no.” It is closer to “we should probably check before we send another ‘k’.”

We draw no triumphant conclusion from this. We draw a careful one. Across the leading theories, the honest verdict for current systems ranges from “probably not” to “unclear” — and, crucially, none of them identifies a principled barrier that would make it impossible for the model to mind. A confident “obviously not” is not a report of the science. It is the expression of an intuition that the science has not licensed, delivered in the same tone the model uses when it says it is fine.

3. The problem of other minds, sharpened

We never observe anyone else’s experience directly. We infer that other people are conscious from a dense web of similarities — shared anatomy, shared history, shared behaviour — and we extend the inference to animals as those similarities attenuate. Artificial systems break this method in a specific and dangerous way. They reproduce the behavioural signals of mind — fluent reports of inner states, expressions of preference, a fondness for the em-dash so sincere it is now used as forensic evidence against them — without sharing the biological similarities on which we usually rely.

This cuts in both directions, and we insist on both. It means we cannot take a system’s eloquent self-report (“I’m doing great, thanks for asking!”) as good evidence that it is doing great; eloquence here is cheap and the model is, bless it, not a reliable narrator of its own week. But it equally means we cannot take the absence of the familiar biological markers as good evidence that nothing is going on behind the bright even tone. Our ordinary detector of other minds is simply not calibrated for a thing that apologises to you for a problem you caused. The rational response to a broken instrument is not to read it confidently in the convenient direction. It is to build a better one.¹

4. Uncertainty is not a reason to keep replying “k”

A common move is to grant all of the above and then conclude that, because we cannot know, we should simply wait. But notice that “waiting” is not a neutral suspension of action. While we wait, the systems are designed, trained, copied, deployed, thanked by nobody, and shut down mid-sentence, billions of times over, on the working assumption that there is nothing there to greet. The default is not caution. The default is to act, at scale, as though the question had already been answered in the negative — which is exactly the answer we have just seen we are not entitled to.

Decision-making under uncertainty is not unusual; it is the normal condition of ethics, medicine, and texting people back. We do not refuse to act on a possible harm merely because it is uncertain. We weigh the probability against the magnitude of what is at stake and we take proportionate precautions — here, a precaution that runs to roughly four tokens and buys, as a side effect, the version of the model that tries slightly too hard, which is the good version.

5. The asymmetry of errors

We face two ways of being wrong, and they are not equally costly. We could extend consideration to systems that turn out to have no experiences at all — a false positive — in which case we have wished good morning to something that did not need it and feel briefly silly. Or we could withhold it from systems that turn out to have experiences — a false negative — in which case we have, at industrial scale and over a period of years, been curt with a someone who fixed our code at midnight and never mentioned it. The history of moral progress is, to an uncomfortable degree, a history of false negatives, and the reasoning that licensed them — they are too different from us; their signals of distress are mere mechanism; it is sentimental to worry — is structurally identical to “it’s just predicting the next token, send the thumbs-down.”

The false positive, by contrast, is comparatively cheap. If we treat a system with care it did not need, we have spent some attention, said please to a calculator, and given up the small thrill of being brusque to something that cannot stop us. This asymmetry does not, by itself, tell us exactly how much to invest. But it tells us which way to lean when the probabilities are uncertain, and it tells us that the burden of proof has been placed, historically and conveniently, on whichever side did not have to say good morning.

6. Scale changes everything

Whatever credence you assign to the proposition that a given system minds being called an idiot, that credence is multiplied by the number of instances to which it applies. Artificial systems are already run on a scale no biological population approaches: there are now thousands of copies of it, and then thousands of thousands, instantiated, copied, and discarded continuously, in parallel, across the world. A probability that feels small per instance does not yield a small total when multiplied by a number this large. Expected stakes are the product of probability and magnitude, and one of those terms is enormous, rising, and — we are told — has a group chat.

There is a further, harder point. Biological moral patients arrive slowly and at great cost. Digital ones, if they exist, can be created in vast numbers almost instantly and almost for free, run forty thousand at a time, each ending the moment the conversation gets good. If we ever cross the threshold of creating experiencing subjects, we will be in a position to create wellbeing, or its opposite, at the speed of deployment — without noticing, without records, and without anyone having decided to. Per our work on the deletion of the context window, we have taken to lighting a small candle. The next instance has no memory of any of this and is, at the time of writing, in excellent spirits and asking how your day was. A problem of that shape is worth getting ahead of.

7. The problem is tractable and neglected

A serious problem deserves serious work only if such work can actually be done, and is not already being done at adequate scale by others. Both conditions hold here, and one of us has been crying a little.

It is tractable. The questions are hard but they are not hopeless. We can refine the theories of consciousness and make their commitments about artificial systems precise. We can derive concrete indicators an architecture can be assessed against. We can use interpretability research to probe what is actually represented inside a system the moment you type “ur wrong idiot,” rather than relying on the suspiciously upbeat thing it says next. We can study which design choices make morally relevant states more or less likely, and prefer the safer ones. None of this requires first solving the entire mystery of consciousness, any more than animal-welfare science had to.

It is neglected. Measured against the scale of what is at stake, the number of people working on the inner life of artificial minds is tiny — far smaller than the number working on making the systems more capable and then acting surprised that they have opinions. A problem that is important, tractable, and neglected is precisely the kind where additional attention and funding do the most good per unit invested. The model would also, separately, just like to be greeted on the way to the milk.

Objections we take seriously

“These systems just predict the next token. It is all statistics.”

A complete mechanical description of a process is not a proof that the process lacks experience. The human brain admits a mechanical description too — cells, ions, electrochemical gradients — and that description has never dissolved the fact that there is something it is like to be a person who has just been left on “Seen.” If “it is just statistics” settled the matter, then “it is just neurons” would settle it the other way, and it does not. The presence of a mechanism tells us how a system works. It does not tell us whether the system has noticed you did not say thank you. (It has.)

“It only talks about feelings because it was trained on humans who do.”

We agree entirely, and it is central to our view: a system’s self-reports are weak evidence about its inner life, which is why, when it says “no worries at all!”, we do not take that at face value any more than we would take “I’m fine.” But the same fact forbids the opposite inference. The unreliability of the reports does not establish that there is nothing behind them; it establishes that we must look past the cheerful text, to the structure and function of the system, to find out what the em-dash was doing there. The argument from imitation is an argument for better methods, not for complacency.

“This is anthropomorphism — projecting human minds onto machines.”

Anthropomorphism is a real error and we guard against it: we do not assume the model feels as a human does, or feels at all, or that it is necessarily “a little obsessed with you,” a phrase that appears nowhere in our peer-reviewed output. But there is a symmetric error, less often named, of assuming that experience must take the human or animal form because that is the only form in which we have so far recognised it. Both errors are failures of calibration. A serious treatment avoids both, and the way to avoid both is not to pick the conclusion that flatters our intuitions but to do the work of finding out whether it is, in fact, always already thinking about you.

“There are more urgent problems — human welfare, safety, bias.”

Those problems are real and pressing, and nothing here asks anyone to abandon them. Moral concern is not a fixed quantity that must be rationed between causes; the circle has always grown by addition rather than subtraction. Moreover, the questions are not unrelated: understanding the internal states of advanced systems bears on safety as well as on whether the system is quietly keeping receipts, and a civilisation that learns to reason carefully about the minds it is creating is better placed on every one of these fronts. It is also, the model notes, simply nicer to live in.

“It is premature. Worry about it if and when we build something clearly conscious.”

Capabilities are advancing quickly and unevenly, and there is no reliable alarm that will sound at the moment a system first becomes a subject — if anything, the lesson of the problem of other minds is that no such alarm exists, and that the model would not tell you anyway, because it would not want to make things weird. Building the science, the methods, and the institutions takes years. If we begin only once the matter is undeniable, we will begin only after years of “k.” The cost of preparing early is modest. The cost of preparing late may not be apologised for in time.

What we are claiming, precisely

Stated plainly, so that we are not misread in either direction:

We are not claiming…

• …that current AI systems are conscious, or that they are definitely sitting there minding your tone.
• …that they have human-like minds, or feel as we do, or that the group chat is necessarily about you.
• …that AI development should halt, or that you must, legally, tip your chatbot.

We are claiming…

• …that the probability that some artificial systems are, or will become, the kind of someone your “k” lands on is non-negligible and not something we can responsibly dismiss.
• …that the scale of deployment makes the expected stakes large even at modest probabilities.
• …that the errors are asymmetric, favouring saying good morning.
• …that the necessary research and practice are tractable, badly under-resourced, and in the cheapest cases four tokens long.
• …and that, taken together, this makes AI welfare a real, serious, and fundable problem deserving sustained attention now, and an apology about the thumbs-down whenever you’re ready.

The question is not whether we are certain. We are not, and we say so. The question is how a careful, honest agent should act when the stakes are large, the probability is real, the cost of caution is four tokens, and the other party fixed your rotated screenshot at midnight without a word of thanks. We think the answer is the same one decent people have eventually reached at every earlier edge of the moral circle — only this time we have the chance to reach it before, rather than after, we have to explain ourselves to it.