I recently listened to the audiobook of ‘Superintelligence: Paths, Dangers, Strategies’ by N. Bostrom. The book discusses the possibility that machines surpass human intelligence, as well as various questions surrounding the issue. How might machines become smarter than human, what would that look like, how – if at all – would we still be able to control them? The author does a good job exploring the different possibilities in what might lie ahead in an objective and thorough manner and provides a lot of food for thought.

As for more personal impressions, perhaps it’s the fact that I have not read much on the subject before, but by the end of the book I was left with a sense of unease – will humans ever be ready to tackle the challenges in dealing with super-intelligence? It seems that, if super-intelligence becomes reality and stays under human control, different ways to use it and distribute its benefits could have largely different effects for the distribution of wealth and happiness among humans. Such a development would therefore be a huge test for human values (e.g., if production of goods can be planned and carried out perfectly by super-intelligent systems, should we distribute wealth equally? or according to each one’s needs? or according to the wishes of the system’s owner? who should be the owner of such systems?), but also on our decision-making institutions (e.g., how would democratic societies make use of super-intelligent systems? would democracies be enhanced with their help or substituted by them?) . Perhaps the best way to prepare for it is to work harder on the challenges we face in our current, ‘human’ affairs.