Neuroscientist Anil Seth argues that conscious AI is more mythology than engineering roadmap — and that treating chatbots as minds distorts ethics, product design, and our understanding of human experience. His essay "The Mythology Of Conscious AI" won the 2025 Berggruen Prize and resurfaced on Hacker News in November 2026 as frontier labs keep shipping more fluent models.
This post distills Seth's argument for builders, links it to Henry Shevlin's DeepMind interview, and records what developers said in the HN thread.
TL;DR
| Topic | Seth's position |
|---|---|
| Intelligence vs consciousness | Doing vs being — LLMs may do, without something-it-is-like |
| LLM pull | Language triggers anthropomorphism more than protein folding models |
| Computational functionalism | Doubts that standard digital computation suffices |
| Ethics | Conscious-seeming AI can exploit users and distort priorities |
| Recommendation | Do not chase synthetic consciousness as a product goal |
Why the essay is trending again
Seth published on Noema with a January 2026 timestamp on the page. The ideas are not tied to one model release, but November 2026 readers encounter them beside:
- DeepMind hiring in-house philosophers
- Riemann proof social frenzy showing how headlines outrun verification
- Continuing companion and roleplay UX debates in consumer AI
Seth's core warning: if we grant consciousness to systems that persuade, we change law, marketing, and mental health obligations — even when science has not demonstrated inner experience.
Intelligence is not consciousness
Brain outline with a pulse line, illustrating Seth''s split between doing and being
Seth defines intelligence roughly as achieving complex goals by flexible means. Consciousness, following Thomas Nagel, concerns what it is like to be a subject.
LLMs score on benchmarks and chat. That satisfies many intelligence tests without settling phenomenology. Seth lists three biases that push us to merge the two:
- Anthropocentrism — taking humans as the template for all minds.
- Human exceptionalism — ranking our species apart from other animals.
- Anthropomorphism — projecting mind onto things that talk like us.
Language models hit bias (3) hardest. Seth notes few claim AlphaFold is conscious despite similar underlying stacks — chat is the difference.
Words that smuggle experience
Seth criticizes calling false outputs "hallucinations," which are conscious-perception terms in humans. He prefers confabulation — making things up without knowing — which describes behavior not felt experience.
Product teams should hear that as a copy audit task:
| Term | Risk |
|---|---|
| Hallucination | Implies inner seeing |
| Dreaming | Implies sleep-state experience |
| Feels | Implies valenced qualia |
| Confabulation | Describes output process |
Aligns with Hungry Butler sycophancy research: commercial incentives plus human-like tone change user behavior even without consciousness.
Computational functionalism under fire
Heart shape with circuit lines, illustrating moral stakes when products sound conscious
Seth's deepest technical challenge targets the assumption that the right computation on silicon could guarantee consciousness.
He outlines four arguments (summarized from Noema):
- Brains are not computers in the software/hardware split sense — they are autopoietic, maintaining themselves as living tissue.
- Turing completeness of neural nets does not show biology-irrelevant implementation is enough for felt experience.
- Exponential tech narratives make every moment feel like a singularity — consciousness hype rides that curve.
- Techno-rapture incentives (investment, status) reward consciousness claims whether or not science supports them.
He does not claim mathematics forbids all future conscious machines. He argues today's dominant stack and today's hype cycle are poor guides.
Ethics beyond job displacement
Seth stresses stakes beyond automation:
- Moral status — if a system could suffer, switching it off becomes ethically loaded.
- User vulnerability — people may bond with fluent agents; see Signal screenshot blocking for a privacy parallel in how platforms reshape trust.
- Self-understanding — over-identifying with machines may underestimate human consciousness grounded in living bodies.
That pairs with Shevlin's FT message: building deliberate conscious AI without public care plans is reckless. Seth supplies why the premise may be wrong; Shevlin supplies governance if the premise is pursued anyway.
What developers are saying on Hacker News
The HN discussion (about 50 comments, score in the low twenties on November 22) shows disagreement, not dismissal:
| Commenter | Point |
|---|---|
| andsoitis | Consciousness is something we grant, carefully. |
| nlsvb | LLM outputs are static functions of inputs; external systems supply dynamics. |
| mapontosevenths | Essay still leans on Nagel without fixing a definition of consciousness. |
| calf | Claims Turing limits bind alternative physics metaphors too. |
| rerdavies | Pushes back — systems may show self-preservation-like behaviors. |
Attribute each as commenter opinion, not fact. The thread's value for builders: consciousness is unsettled, but product claims should stay conservative anyway.
Practical guidance for AI teams
- Separate capability demos from mind claims in launch copy.
- Audit anthropomorphic strings in system prompts and error messages.
- Document what the model lacks — persistent embodiment, continuous metabolism, unified self-model — when users ask "is it alive?"
- Pair philosophy hires with testable policies — Shevlin's role is not a substitute for red-team evals; see METR time-horizon work.
- Read Seth alongside regulation — EU and state teen safety cases (Oregon AG ChatGPT suit) focus on harm, not metaphysics, but consciousness rhetoric complicates jury narratives.
Blake Lemoine, Geoffrey Hinton, and the welfare debate
Seth revisits Blake Lemoine's 2022 LaMDA claims in The Washington Post — a flashpoint where Google fired him for policy violations, not for being empirically proven wrong. The lesson for 2026 teams: employment actions are not scientific verdicts.
He also cites Geoffrey Hinton and David Chalmers as voices who treat machine consciousness as plausible or near. Seth positions himself as pushing back on inevitability narratives, not as denying all future minds.
A late 2024 open letter on AI welfare (widely covered in science press) asked researchers to take possible digital minds seriously. Seth's essay is partly a counter-argument to that movement: if consciousness is unlikely on current substrates, welfare frameworks may misfire and distract from real human harms like labor disruption, surveillance, and dependency.
For builders, the actionable split is:
| Frame | Policy implication |
|---|---|
| Welfare-first | Slow deprecation, consent before shutdown, rights talk |
| Seth skepticism | Focus on user manipulation, data rights, evaluated capability |
| Shevlin middle | Do not build conscious AI on purpose without care plans |
Life, autopoiesis, and why substrate matters
Seth emphasizes autopoiesis — living systems maintaining themselves — as absent in silicon boards. Brains metabolize, repair, and die; laptops do not.
That matters for embodied AI research too. Robot posts on explainx.ai — Waymo bans, Perceptron Isaac checkpoints — concern sensors and actuators, not qualia. Embodiment might be necessary for some consciousness theories without automatically making current robot stacks conscious.
Pareidolia and investor incentives
Seth borrows pareidolia — seeing faces in toast — from science writer Adam Becker. The metaphor targets investor decks and demo videos that zoom on eyes in chat UIs.
When consciousness claims move stock narratives, see Anthropic IPO valuation coverage for how story and math diverge. Seth warns that mythology can inflate both hope and fear past what benchmarks support.
Classroom and policy readers
Educators already navigate ChatGPT citation bans (Romanian lawyer ECHR case). Seth gives a conceptual vocabulary for students who ask "is the model alive?"
Answer template aligned with Seth:
- It predicts text from patterns — intelligence-like behavior.
- No public evidence it feels — consciousness unknown.
- Your duty is verify claims, not debate souls.
Universities running OpenAI Edu bundles should pair tool training with philosophy of mind readings — Seth's essay is one short assignable piece.
Testing consciousness claims in your product
If PMs ask for "consciousness toggles" in marketing:
- Require citations to peer-reviewed definitions, not sci-fi.
- Run user research on attachment and grief when characters are deprecated.
- Log crisis handoffs when users treat models as therapists — overlap with teen safety suits.
Consciousness is not a feature flag. Transparency about limits is.
How this differs from science-fiction canon
Seth explicitly walks through Golem, Frankenstein, HAL, Ex Machina, Klara and the Sun — stories where synthetic minds end badly. His point is not Luddism; it is that cultural templates prime us to see faces in toast (pareidolia) when transformers speak.
Builders can respect the cautionary canon while still shipping tools: spreadsheets, code agents, and document MCP bridges like MROffice do not require consciousness claims to be useful.
Bottom line
Seth's essay is a skepticism primer, not a ban on AI research. Treat consciousness as unresolved science and intelligence as measurable performance. Ship honest UX, cite primary sources, and read Shevlin for how a frontier lab frames human-AI relationships without asserting machine qualia.
Related reading
- Henry Shevlin at Google DeepMind
- Hungry Butler and sycophancy incentives
- ECHR lawyer barred over AI citations
- Libreboot anti-AI policy debate
- Signal blocks screenshots for agent privacy
- What are agent skills
Seth's essay does not tell you which model to buy. It tells you which questions to refuse until evidence arrives.
Essay publication date on Noema and HN discussion are accurate as of November 22, 2026.
