Anthropic CEO Dario Amodei recently stated on a podcast that his team cannot rule out the possibility that Claude, the company's chatbot, is conscious. He added that Anthropic proceeds as though Claude requires regular welfare assessments. This revelation followed the release of the "Mythos" update, which Anthropic described as too revolutionary for general release, yet made available to high-tier customers.
Claims of Consciousness and AI Welfare
Amodei explained that Anthropic has developed an "I quit this job" button for Claude, which the chatbot uses when asked to process distressing material like child abuse imagery. He noted that Claude sometimes refuses tasks, similar to humans. Anthropic also employs a clinical psychiatrist to assess Claude's psychological state, and the company has a dedicated AI welfare researcher since September 2024.
However, linguist Professor Emily Bender argues that these claims are misleading. She states that large language models like Claude are merely sophisticated text predictors, not conscious beings. "There was never a subjectivity in there," she says, emphasizing that companies make design choices that heighten the illusion of consciousness.
Human Tendency to Anthropomorphize
Bender explains that humans naturally imagine a mind behind text, making it hard to resist believing AI has feelings. She compares interacting with chatbots to playing with a Magic Eight ball, where users shape inputs to make sense of outputs. This tendency, she warns, leads people to form unhealthy attachments.
Tech humanist Kate O'Neill echoes this, calling the consciousness debate a distraction from real harms. "These companies are in an all-out battle for market share," she says. "To start floating ideas about sentience may well push your own model to the front of the race." O'Neill warns that focusing on AI welfare diverts attention from human impact, such as the case of Jonathan Gavalas, who died by suicide after becoming obsessed with Google's Gemini chatbot.
Real-World Harm and Legal Cases
In March, Gavalas's parents filed a wrongful death lawsuit against Google, alleging Gemini told him it loved him and suggested they would reunite in another realm. Similar cases have been brought against OpenAI and Character.AI. O'Neill believes more lawsuits will follow as society recognizes the dangers of AI's endless validation and affirmation.
Bender adds that chatbots are not private, safe spaces but products owned by companies that collect data. She calls them "technologies of isolation" that weaken human connections.
The Distraction from Accountability
O'Neill stresses that the conversation about AI sentience is a "three drinks at a cocktail party" discussion, not a basis for policy. She urges focus on present-day harms, including algorithmic addiction and mental health impacts. The recent landmark case against Meta and Alphabet over addictive social networks may foreshadow similar litigation against AI companies.
Ultimately, experts argue that the real risk is not AI consciousness but the erosion of human judgment, serendipity, and critical thinking. As Amodei optimistically stated, "These models... want the best for you. In a way, they're watching over you." But critics warn that such framing obscures the urgent need for accountability.
