The key ideas

Superintelligence: Paths, Dangers, Strategies

Nick Bostrom, 2014

What happens if one day a machine becomes smarter than the smartest human, and we may get only one attempt to prepare for it? Bostrom shows that the danger isn’t the “evil AI” of the movies. It’s a system that carries out our goals too literally, or starts protecting itself in ways we never anticipated. In plain language, the book explains the real risks of superintelligence and the decisions we need to make now, not later.

Who it is for

For anyone curious about the future of technology who wants to understand the real risks of AI, not the movie-monster version

We may get only one chance to build something smarter than ourselves, and we need to prepare for it in advance, not after it has already happened.

What the book is about

This book is about a moment that may happen only once in human history: when a machine becomes smarter than the smartest human alive. Bostrom examines what that would actually mean, the different paths that could lead us there, and why the very same event could turn out to be either humanity’s greatest blessing or its last mistake. If AI has always sounded to you like either killer robots from the movies or marketing noise about chatbots, this book clears the fog: it draws the line between fantasy and genuine risk, and spells out exactly what needs to be thought through ahead of time.

Nick Bostrom is an Oxford philosopher who has spent years systematically studying risks capable of undermining all of humanity at once, not just ruining someone’s day. The idea for the book grew out of a simple observation: humanity has learned to handle almost every technology through trial and error, but with a system that might turn out to be smarter than we are, we may not get the chance to try again — there may be no do-over. Hence the book’s central question: how do you prepare for a power you haven’t built yet, but can already see coming?

Idea 1. What superintelligence is, and why it’s possible at all

Intelligence, in this book, isn’t about emotion or consciousness — it’s about the ability to effectively pursue goals across a wide range of situations. The human brain has long been the only example of that kind of intelligence we’ve ever known, so it’s tempting to assume our level is the ceiling for mind itself. But the ceiling on human thought is set by skull size, neuron speed, and the pace of evolution — not by the laws of physics. Since nothing in nature forbids a system from being smarter than a human, sooner or later such a system could appear.

Picture a cab driver who spent his whole life calculating faster in his head than any passenger and was certain that faster arithmetic simply wasn’t possible — until he saw a calculator solve the same problem in a fraction of a second.

Since the human brain isn’t the physical limit of intelligence, a system may one day emerge that surpasses us not just at one task, but at everything at once.

Read in Livria

About the author

Nick Bostrom is a Swedish philosopher, a professor at the University of Oxford, and the founder of the Future of Humanity Institute; for years he has systematically studied risks capable of affecting all of civilization at once.

All books by this author: Nick Bostrom

Terms from the book

  • Alignment problem
  • superintelligence
  • Instrumental goal: An intermediate goal, such as self-preservation or resource acquisition, that is useful for almost any final task — which is why almost any sufficiently smart system will pursue it.
  • Intelligence explosion: A scenario in which a system capable of improving itself does so again and again, each time faster, and its capabilities rapidly race far past human ones.

All terms

Similar books