On the myth of AI consciousness - and what it means

A great science-fiction writer blows away the farfetched (but hugely profitable) fantasy that artificial intelligence engines have genuine insight into themselves - or existence.

Claude may have some functional version of emotions or feelings.

This is how Anthropic, the five-year-old, $1 trillion (yes, trillion) company that created the popular artificial intelligence model it calls “Claude,” describes its product.

Spoiler alert: No, “Claude” does not have, cannot have, and will never have emotions or feelings. What Claude has is the ability to produce a written simulacrum of emotions and feelings. And good marketing. Very good marketing.

—

(Human-written. Now and forever. With your help.)

—

Ted Chiang, the best science-fiction writer of our time, made this case brilliantly in a (paywalled) piece in the Atlantic on Wednesday.¹

Chiang starts by considering the case where a user asks an artificial intelligence engine (or, more accurately, a “large language model”) to imagine a conversation between Julius Caesar and Genghis Khan. The person reading the story the engine wrote would not think the imagined Khan was alive or had feelings, no matter how real he and his words seemed on the screen.

Chiang then points out if we asked the engine to produce a conversation between an imagined user of its services and itself, it would do the same, with similar fluency. Again, though, we would know we were seeing a work of fiction, an onscreen novel with imagined characters. We would not consider either character alive.

But somehow, AI companies — particularly Anthropic — have convinced us that a similar conversation between a human user and a software engine may indicate the engine is conscious. As Chiang writes:

The human might form a powerful impression that she’s conversing with a conscious entity, but she is not; she’s interacting with a character precisely as fictional as the Julius Caesar or Genghis Khan characters in the earlier example.

—

(Open the pod bay doors, Hal.)

—

Chiang goes on to explain no matter how good these engines get at claiming consciousness or producing writing that stirs human emotion, they will never be anything more than what they are.

The perfection of the process or its output does not matter. To use an analogy Chiang does not, recordings were once scratchy and lasted seconds. Today they reproduce voices and concerts and all sounds with essentially complete accuracy. No matter, they are still recordings, not originals.

Chiang explains what he would want to see to be convinced Claude or another “large language model” had actual consciousness. “The first requirement is that the computer program has a body (either physical or virtual) and sense organs,” he notes. “Without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.”

He offers a long list of potential requirements, none of which the models have met or are on any conceivable path to meeting.

In sum, Chiang argues Claude has no ability to feel, merely the ability to claim feelings — based on its very real ability to break the endless information on the Internet into bits of data and process them for patterns, then regurgitate those patterns to us in superficially novel ways.

Further, he notes that despite their claims the models are or at least may be conscious, Anthropic and other AI companies do not treat their products that way. On the most basic level, they let users turn off the models — and they themselves retire models (sometimes even over the objection of human users). As Chiang writes:

If the participants in a conversational transcript had any [consciousness], you would have some duty to extend the transcript to prolong their existences; merely keeping a copy of Microsoft Word 2010 backed up on a USB stick isn’t going to help them.

This is not quite checkmate, but it’s close.

—

Want to support writing like this but not ready to subscribe? Try a one-time donation!

—

As good as Chiang’s piece is, though, he hardly touches on the underlying reason humans are so quick to ascribe consciousness to lines of code that exist only in massive datacenters — a willingness Anthropic and other companies have exploited brilliantly.

Human beings can never truly know whether anyone else is conscious, much less what consciousness feels like to anyone else.

Anyone else includes everyone else: our closest friends, our spouses, our parents, even our children — whom we have created. We simply cannot prove other people are not “non-player characters,” as gamers say. But this fact predates computer simulations of reality. At other times and in other places, these ersatz humans have been called robots, or zombies, or djinn.

The philosophical implications of this fact are profound and terrifying. For our own mental health we all (wisely) ignore it nearly all the time. Without considering the issue too closely, we simply assume other people are conscious.

After all, other humans look like us, they move through the world in the same way we do, they also need to eat and drink to survive, they also have have eyes and ears to see and hear the world and brains to process it. And they have the same emotions that we do: the same awe at a beautiful sunset, the same love for their spouses or children, the same mortal fears.

How do we know? Because they tell us so, in language that we understand.

It is far easier, and safer, to believe other humans are conscious in the same way we are — and to hope they are extending the same courtesy to us.

We should all be glad for this reciprocal consideration (and doubly glad we don’t think about it too often). But it means human beings have a general bias toward extending the presumption of consciousness outward toward nonhuman beings.

—

This tendency is a subset, the most crucial subset, of the broader trait called anthropomorphism — interpreting nonhuman things or events in human ways: it’s hasn’t rained this year because the gods are angry. I can tell my dog about my fight with my boss, she understands. My car knows where I’m going before I do.

In 2002, a classic television ad from IKEA satirized this tendency brilliantly. The 60-second spot shows an old red lamp being taken out of an apartment and put on a city street alongside a bag of trash as mournful music plays.

Night falls, and rain pours down on the sad red lamp. Inside the warm apartment a new lamp — the replacement for this sad lost fellow — shines brightly, until finally its owner turns it off.

Finally, a man enters the frame and walks toward the camera. “Many of you feel bad for this lamp,” the man says in an odd Mitteleuropa accent.

“That’s because you crazy. It has no feelings. And the new one is much better.”

Of course, the viewer thinks, as IKEA’s blue-and-yellow logo fills the screen. Of course he’s right! It’s a lamp. Having nostalgia for it, fine, that’s a me feeling. But feeling bad for this piece of metal and wires, that’s insane.

IKEA wanted viewers to remember that reality, so they would be happy to replace their old junk from IKEA with new, “much better,” junk from IKEA. But Anthropic (such a revealing name) and the other AI companies have literally trillions of dollars at stake in their own version of this story, one in which the lamp has all the feels.

—

(Don’t be sad because the light is out. Be happy the light once shonr.)

SOURCE

—

Ted Chiang doesn’t like Anthropic or the AI companies much. Late in the piece, he writes:

Many people feel LLMs are a fundamentally unethical technology because they are built on the theft of intellectual property, rely on exploited labor, waste natural resources, spread misinformation, deskill workers, stunt the cognitive development of students, and contribute to a consolidation of power that is unhealthy for a democratic society.

One doesn’t have to go that far to recognize that the AI companies have become surprisingly important to both national governments and financial markets in record time. No company embodies this rise more quickly than Anthropic, which plans to sell shares to the public later this year at a valuation of over $1 trillion — potentially making more valuable than any other company outside the technology industry except for Saudi Aramco.

Again, Anthropic did not even exist in 2020.

—

(If you like this work, please support it.)

—

Anthropic and the other AI engines are without doubt extraordinarily powerful tools for the creation and manipulation of language in all its forms, including software coding and math, which are both language. But so far, they have not managed to make themselves useful in ways consumers will pay for. Their value to companies is also highly speculative. Nor have they meaningfully differentiated the engines by creative output, although coders seem to prefer Anthropic’s Claude. All the engines use the same underlying “token weighting” process to map the information they are examining.

It is worth wondering if these companies would command these valuations and attention if their products were called “artificial coding and language tools” or “predictive and robotics control software” instead of “artificial intelligence.”

Raising the question of whether they are conscious, as the companies have done. makes them more novel and exciting still. And could they be conscious?

The question seems plausible at first. After all, like humans, the models are telling us so, in language we understand.

But lamps can’t be conscious of the light they make.

Neither can Claude.

1. I’ve already discussed my feelings about the Atlantic, I’m not going to repeat them.