Superintelligence: Why the Idea That Eats Smart People Won't Let Go

In 2016, the programmer and essayist Maciej Cegłowski gave a talk in San Francisco with a title that has aged uncannily well: Superintelligence: The Idea That Eats Smart People. His argument, delivered to a room of engineers, was disarmingly simple. There exists a class of ideas so seductive that they reliably hijack the minds of otherwise brilliant people — and the belief that humanity is on the verge of building a godlike artificial intelligence is one of them.

Nearly a decade on, with ChatGPT in every browser and Sam Altman warning Congress about existential risk, Cegłowski's diagnosis deserves another look. Not because superintelligence is impossible — nobody can prove a negative about the future — but because the shape of the belief, and the people it most reliably captures, tells us something important about how technology gets discussed, funded, and feared.

What the superintelligence thesis actually claims

The canonical version, popularised by Oxford philosopher Nick Bostrom in his 2014 book Superintelligence, runs roughly like this. At some point, we will build an AI smart enough to improve its own code. That improved AI will be smarter still, and able to improve itself faster. Within a short period — months, days, perhaps minutes — this recursive self-improvement produces an entity vastly more intelligent than any human. Because intelligence is, in this framing, the master skill that unlocks all others, the superintelligence becomes effectively omnipotent. And because we have no reliable way to specify its goals, it will probably pursue something other than human flourishing, with consequences ranging from inconvenient to apocalyptic.

From this chain of premises follows the entire ecosystem of AI safety research, existential-risk philanthropy, and the recurring public letters signed by tech billionaires asking the world to pause. The argument has the elegance of a syllogism. It is also, when you poke at the joints, more fragile than its evangelists let on.

Why smart people find it irresistible

Cegłowski's central observation was sociological rather than technical. The superintelligence story flatters its believers in several specific ways, and the people who staff Silicon Valley are unusually susceptible to that flattery.

First, it places intelligence — the trait its proponents possess in abundance — at the centre of the universe. If intelligence is the lever that moves all other levers, then those who have it most are the natural priests of the coming age. Second, the story is grand. It offers participation in a cosmic drama on the scale of religious eschatology: heaven (a benevolent AI ushers in post-scarcity utopia) or hell (paperclip maximisers turn the solar system into office supplies). Third, it is actionable only by insiders. The general public cannot meaningfully contribute; only a small cadre of philosophically inclined engineers can steer the outcome.

This is, structurally, the same appeal that draws people to millenarian movements throughout history. The dressing is mathematical rather than scriptural, but the underlying offer — you, specifically, are positioned to save or doom the world — is identical.

The premises that don't quite hold

The technical objections to the standard superintelligence narrative are well known to specialists but rarely make it into the public conversation. A few are worth dwelling on.

None of this means current AI systems are safe, or that scaling them is wise, or that they cannot cause enormous harm. It means the harms worth worrying about look very different from a Bostromian singleton.

What the obsession crowds out

The real cost of the superintelligence frame is what it pushes off the agenda. While billions of dollars and the attention of the smartest people in machine learning flow toward speculative alignment problems involving hypothetical agents, the concrete harms of actually existing AI are comparatively under-resourced.

For an Australian audience this matters in tangible ways. The Robodebt scheme — an automated debt-recovery system run by Centrelink that wrongly pursued hundreds of thousands of welfare recipients — was not a superintelligence. It was a mediocre algorithm wrapped in bad policy and worse oversight. The royal commission that followed found that the harm was caused by the very human failure to ask whether a statistical inference was an adequate basis for legal action. No paperclip maximiser required.

Similar patterns recur globally: facial recognition systems with racial accuracy gaps deployed in policing, generative models trained on copyrighted work without consent, large language models hallucinating citations in court filings, hiring algorithms reproducing the biases of their training data. These are the AI risks most people will actually encounter. They are boring, regulatory, and tractable. They do not flatter the people building the systems by making them protagonists in a cosmic drama, which may be part of why the field's loudest voices prefer to talk about something else.

The political function of existential risk

There is a final, more cynical reading of why superintelligence dominates the discourse. Talking about existential risk is, conveniently, a way for AI labs to signal extraordinary capability while requesting regulation that only they could possibly comply with. Our product is so powerful it might end the world; therefore only we, the responsible ones, should be allowed to build it. The narrative is simultaneously a marketing pitch and a moat.

This is not necessarily conscious cynicism. The people warning loudest about AI doom mostly seem to believe what they are saying. But beliefs that happen to align with one's commercial interests deserve sharper scrutiny than beliefs that don't, and the superintelligence consensus inside frontier labs has rarely received that scrutiny from within.

Holding two ideas at once

The honest position is uncomfortable. AI systems will become more capable. Some of those capabilities will be dangerous in ways we cannot fully anticipate. It is reasonable to fund research into making them more controllable and transparent. None of this requires accepting the specific package of claims bundled under the word "superintelligence," and accepting that package wholesale has costs.

Cegłowski's title was deliberately rude. He chose eats rather than fascinates because he thought the idea was metabolising the people who held it — turning capable engineers into theologians, capable philanthropists into doomsday preppers, capable companies into priesthoods. A decade later, watching tech founders speak in increasingly apocalyptic registers about their own products, that verb still seems exactly right.

The most useful thing the rest of us can do is decline to be eaten. Take AI seriously. Take its current harms more seriously than its hypothetical ones. And reserve a healthy scepticism for any story in which the smartest people in the room turn out to also be the most important people in human history. That has, historically, almost never been true.

Related on Bleen

Sources