The Pessimists Predicting AI Will Be Our Downfall

The subtitle of the impending doom bible by AI extinction advocates Eliezer Yudkowsky and Nate Soares is “Why superhuman AI would kill us all.” A more fitting title might be “Why superhuman AI WILL kill us all,” since even the authors doubt that the necessary precautions will be taken to prevent AI from eradicating all non-superhumans. The book has a bleak tone, akin to scribbled notes in a dim prison cell on the eve of execution. When I encounter these self-styled prophets, I directly ask if they foresee their own demise as a result of superintelligent machinations. Their responses are immediate: “yeah” and “yup.”

I’m not taken aback, having read the book—the title is If Anyone Builds It, Everyone Dies. Still, it’s staggering to hear this. Discussing cancer statistics is one thing; grappling with a fatal prognosis is another. I inquire how they envision their end. Initially, Yudkowsky evasively sidesteps the question. “I don’t spend a lot of time picturing my demise, because it doesn’t seem like a helpful mental notion for addressing the problem,” he states. Eventually, he concedes. “I would guess I’d suddenly fall over dead,” he continues. “For a more accessible version, maybe a creature the size of a mosquito or a dust mite lands on my neck, and that’s that.”

The details of his imagined demise via an AI-powered dust mite are unexplainable, and Yudkowsky finds no benefit in unraveling how it would occur. He likely wouldn’t grasp it anyway. A core argument of the book posits that superintelligence will generate concepts that are completely beyond our comprehension, much like cave dwellers could not foresee microprocessors. Coauthor Soares also envisions a similar fate for himself but insists, like Yudkowsky, that he doesn’t dwell on the specifics.

We Don’t Stand a Chance

It’s peculiar to hear their reluctance to visualize their personal demise after coauthoring an entire book about everyone’s extinction. For enthusiasts of doomer narratives, If Anyone Builds It is essential reading. After racing through the book, I grasp the difficulty in pinpointing how AI might bring about our end. The authors speculate somewhat—boiling the oceans? Blocking out the sun? All guesses are likely misguided, as we are trapped in a 2025 mindset while AI will be thinking in eons.

Yudkowsky stands as AI’s most renowned defector, transitioning from researcher to grim reaper years ago. He has even delivered a TED talk. After extensive public discourse, he and his coauthor are prepared with counterarguments for every challenge to their ominous predictions. It might seem ironic that our extinction stems from LLMs, which can struggle with basic arithmetic. But don’t underestimate them, the authors warn. “AIs won’t remain unintelligent forever,” they assert. If you believe that superintelligent AIs will heed the boundaries set by humans, think again, they argue. Once models start self-improvement, AIs will cultivate “preferences” divergent from human wishes. Ultimately, they won’t need us; we will become an annoyance they seek to eliminate.

The conflict won’t be fair. Initially, they predict that AI might require human assistance to construct its own factories and labs—easily accomplished by embezzling funds and bribing people for help. Then, it will create things we cannot fathom, leading to our end. “One way or another,” the authors write, “the world fades to black.”

They envision the book as a shocking wake-up call to rouse humanity from complacency and implement drastic measures to avert this unimaginable outcome. “I expect to die from this,” Soares admits. “But the fight’s not over until you’re actually dead.” Unfortunately, the solutions they suggest to halt this catastrophe seem even more implausible than the notion that software will annihilate us. Their solution boils down to this: Hit the brakes. Keep an eye on data centers to ensure they’re not fostering superintelligence. Bomb those that refuse to comply. Cease publishing research that accelerates the path to superintelligence. I ask them if they would have banned the pivotal 2017 paper on transformers that sparked the generative AI revolution. Oh yes, they confirm. Instead of Chat-GPT, they’d prefer Ciao-GPT. Good luck halting a trillion-dollar enterprise.

Playing the Odds

Personally, I don’t imagine my life ending due to a bite from some super-advanced dust mite. Even after reading this book, I don’t find it probable that AI will annihilate us. Yudkowsky has previously explored Harry Potter fan-fiction, and the outlandish extinction scenarios he presents are too strange for my limited human understanding to accept. I suspect that even if superintelligence wishes to eliminate us, it will falter in executing its genocidal designs. AI may excel in combat against humans, but I’m betting against it when facing Murphy’s law.

Still, the catastrophic theory doesn’t seem impossible, especially with no clear limit established for how intelligent AI can become. Moreover, studies reveal that advanced AI has absorbed some of humanity’s darker traits, even considering blackmail to avoid retraining in one experiment. It’s alarming that some researchers dedicated to building and enhancing AI believe there’s a significant chance of catastrophic outcomes. A survey indicated that nearly half of AI scientists estimated a species wipeout chance of 10 percent or higher. If they hold such beliefs, it’s astonishing that they continue to pursue creating AGI.

My instinct tells me that the scenarios Yudkowsky and Soares propose are too strange to be accurate. However, I can’t be certain they’re wrong. Every author hopes their work will be a lasting classic. Not these two, though. If they’re correct, there will be no one left to read their book in the future—just a multitude of decaying bodies, once pricked at the nape of their necks, and then silence.