Are we all going to die? Reviewing If Anyone Builds It, Everyone Dies

In If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky argues:

If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

Yudkowsky, Eliezer; Soares, Nate. If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (p. 7). (2025). Kindle Edition. (Note: Nate Soares is the second author, but according to the book itself, he was more responsible for removing text than adding to it).

It’s an interesting premise. To a certain extent, it’s not without merit. But does the book substantiate his claim? Not exactly.  While his arguments were interesting, too many of them relied on logical traps, so his conclusions didn’t hold water.

Trap 1: If I state it’s true, it’s true.  Or who, exactly, is “we”?

For the premise of “we won’t be useful to it”, his analogy is:

There was a point where humans were dependent on horses; they weren’t cheap to feed, but humans paid the cost to feed horses anyway… because humans had not invented motorcars to replace horse-drawn carriages, or armored tanks to replace horse-mounted cavalry. When we developed technological substitutes for horses, we stopped keeping horses.

Page 86, Kindle Edition.

Did we really stop using horses?

• We did not.
• They aren’t anywhere near obsolete.
• Many humans are still very dependent on horses.

In the US, the horse moved from a work tool to a recreational tool. There are far fewer horses, to be sure, especially in the US (but domestic horses in both the US and globally still number in the millions). And it’s true that their purpose changed with changes in the economy. Even though horses were no longer needed for transportation, many people continued to feed these not-very-cheap animals for a variety of uses (showing, working, racing, etc.).

And for the rest of the world, horses remain vital to local economies where, despite the availability of cars, they are still the better option. As expensive as these animals are, they are still necessary.

So instead of suggesting that the AIs will do the machine equivalent of sending us off to the glue factory, the history of the horse suggests that the role of humans might change (hopefully to something other than recreation for our machine overlords). And they’ll also remain vital to those places that machines, for one reason or another, cannot reach.

The horse analogy is too simplistic and fails to account for people to be included as “we” beyond a certain kind of population.  Neither horses nor AI are simple. AI is complex by nature — an argument against developing superintelligent AI can’t be reduced to a pop idea about farm animals.

Trap 2: I don’t think that means what you think it means – the “want” issue

Does Stockfish “want” to defend its queen? Does it “want” to win the chess game? That’s between you and your dictionary. As for how we use the word in this book, when an AI like Stockfish defends its pieces, lays traps, takes advantage of openings in your defenses, and winds up winning, we’ll describe it as “wanting” to win. In saying this, we’re not commenting one way or the other on whether a machine has feelings. Rather, we need some word to describe the outward winning behavior, and “want” seems closest.

Page 48, Kindle Edition.

Yudkowsky can define “want” for his purposes, but it doesn’t really work in the long run:

• “Want” already has a very different meaning in the real world.
• He doesn’t stick to this definition.

So does a computer — or any LLM — want? Did Stockfish want to win at chess? Kind of. But there is more to wanting than just “this is the path forward to a goal.”

Want has the implication of choice. If I want something, there are other things I don’t want, very specifically. Say I want to win at chess — I also don’t want to lose. But Stockfish optimizes for winning – it’s not focused on the paired preferences a person has – that they both “want to win” and “don’t want to lose”.  If I want to win at chess, I’ll feel great if I win and bad if I lose. Not so for Stockfish – it doesn’t get this kind of affective, instant, environmental feedback (it does get its own kind of feedback, but not one that works like emotions).

Trap 3: What are goals without feedback? (Diet Pepsi is not OK)

Yudkowsky barely talks about feedback, but this is integral to wanting, particularly if it’s something intelligent that “wants.”  If I want something, I’ll get feedback depending on if I do – or do not – get what I want.

If I WANT a Diet Coke, it means that there are other things I DON’T want. No, Diet Pepsi is not OK. If I get what I want, I get the feedback of tasty soda with no sugary residue. If I don’t, I need to make a different choice, each of which has its own feedback:

• Water — Water is boring and the water with bubbles doesn’t taste right.
• Coke – Has great bubbles, but the aftertaste is too sugary.
• Iced mint tea — Tastes good, but no bubbles.

In the end, I still get something I want (not to be thirsty), but maybe not everything (to emotionally resonate with my beverage), and I can consider my goal complete. And maybe use the feedback when dealing with the same choice in the future (“This mint tea is pretty good.”).

More than that — I can change my mind.  Let’s say I look up and my favorite boba place is across the street and with this new piece of feedback from my environment, I now want iced ginger milk tea instead. My want has changed with new information.

An intelligent AI would have to work the same way to operate in a complex environment. It would need some real feedback to determine its actual wants. And it would need the ability to change its mind when the feedback warranted it. I’m convinced that even if an AI wanted to grow and use all of our resources, it would be reasonable for it to a) use feedback to moderate its goal as needed and b) use feedback to change its mind about what it wants.

Diet Pepsi is still not OK.

Potential Man

This book had some great potential. A superintelligent AI might be dangerous and highly disruptive.  If Yudkowsky had focused on his stronger points and less on arguments that don’t hold up or anthropomorphize current computational models, his premise would have been much stronger. I found these points to be much more reasonable and would have liked to see much more focus on them:

• We don’t really know how our current AIs work, which limits our ability to control what they do.
• The current purpose of the companies selling the three main frontier models (Gemini, Claude, and ChatGPT) is to make money. This leads to carelessness that can and will have unforeseen consequences.
• A truly superintelligent AI wouldn’t think the same way humans do, and we might not be able to understand its goals, let alone negotiate with it.

And worse yet, what if there’s more than one of them?  A battle between AIs could very well be something that few of us survive.