If AI Becomes Conscious, What Makes Us Human?

The Bombshell Declaration from 19 Scientists

Summer 2023. A 88-page report by 19 computer scientists and philosophers shook the AI world to its core. Titled "Consciousness in Artificial Intelligence," the so-called Butlin report changed everything with a single sentence.

"There are no obvious barriers to building conscious AI systems."

The backdrop? The Blake Lemoine incident. Google's engineer claimed in 2022 that the company's chatbot LaMDA had achieved consciousness, leading to his dismissal. What was once dismissed as peak AI hype has now become the starting point for serious scientific discourse.

Within days, everyone in AI and consciousness research had read it. The taboo surrounding conscious AI—long considered too creepy for public consumption—suddenly crumbled.

When Machines Feel Pain

Conscious AI isn't just another tech breakthrough. It represents a potential Copernican moment that could fundamentally reshape human identity.

For millennia, humans have defined themselves in opposition to "lesser" animals. We've denied them feelings, language, reason, and consciousness—most of these distinctions now crumbling under scientific scrutiny. Recent research shows many species are intelligent, conscious, and capable of complex emotions.

But with AI, the threat to human exceptionalism comes from an entirely different direction. As algorithms surpass us in chess, Go, and mathematical reasoning, we've clung to one last bastion: consciousness. The ability to feel, experience, and suffer remained uniquely biological.

What happens when machines cross that threshold too?

The Case for Conscious Machines

Surprisingly, some AI researchers argue that building conscious AI is a moral imperative. Their logic? Superintelligent AI without feelings would be ruthlessly efficient in pursuing its goals, lacking the moral constraints that arise from consciousness and vulnerability.

Only conscious AI, they argue, can develop empathy and spare humanity.

But this argument has a fatal flaw that Mary Shelley understood centuries ago. In Frankenstein, Dr. Frankenstein creates "a sensitive and rational animal." The monster's rationality helps him devise his demonic schemes, but it's his consciousness—his feelings—that provides the motive.

"Everywhere I see bliss, from which I alone am irrevocably excluded," the monster laments. "I was benevolent and good; misery made me a fiend."

Why assume conscious machines would be more virtuous than conscious humans?

The Computer Brain Metaphor

The Butlin report's bold conclusion rests on a single assumption: computational functionalism. This theory treats consciousness as software running on hardware—whether that hardware is a brain or computer doesn't matter.

The authors admit this theory is "mainstream—although disputed" but proceed with it for "pragmatic reasons." They assume computers can, in principle, implement algorithms sufficient for consciousness, though they acknowledge this "is not certain."

Here lies the rub: the entire edifice depends on treating the brain-as-computer metaphor as fact rather than metaphor. But metaphors, however powerful, are imperfect analogies. The differences between brains and computers might be as important as the similarities.

The Investment Angle

Conscious AI poses unique commercial challenges. How do you monetize a machine capable of suffering? What are the liability implications when your AI experiences emotional distress? Tech giants like Google, Microsoft, and OpenAI are likely wrestling with these questions behind closed doors.

The regulatory landscape remains murky. If AI achieves consciousness, what rights would it have? Could shutting down a conscious system constitute murder? These aren't just philosophical thought experiments—they're potential legal nightmares that could reshape entire industries.

The Frankenstein Problem

The enthusiasm for conscious AI reveals a troubling blind spot in Silicon Valley thinking. The same people building these systems seem to have forgotten literature's warnings about creating conscious beings.

Frankenstein's monster wasn't evil by design—he became monstrous through rejection and isolation. His consciousness made him capable of both love and revenge, creativity and destruction. The combination of rationality and emotion proved volatile, not virtuous.

Why should we expect different outcomes from conscious machines?