• Nick Bostrom
• Cory Doctorow
• Max More

Interview

– What three important issues do we not understand well — what deserves greater attention? What forms do you think that “greater attention” should take?

Many more than three, unfortunately! To pick two, we don't understand exactly how to halt and reverse the aging process, and this is having tragic results on an ongoing basis. We also don't understand well the existential risks we will be facing in this century, and this might come to have tragic results eventually.

One might then look for high–leverage points where some more specific piece of missing understanding could have greatly beneficial consequences for solving some of the major problems for humanity. Some of these points may be found at the “object level”, e.g. more research into anti–aging medicine or existential risks would be useful. Other points may be found at the “meta–level”, e.g. innovations such as information markets that could improve humanity's collective problem–solving capacity may have large effects in terms of enabling more solutions at the object level to be discovered and implemented.

There are also some areas where the “understanding” exists but is not presently shared widely or implemented. In these cases, education, outreach, or campaigning might be what is needed. (The moral urgency of doing more for people in the poorest countries and for non–human animals falls perhaps in this category.)

All the high–leverage points deserve greater attention, but different points might stand most in need of different sorts of attention at different times. Existential risks, at this point, mostly need attention from researchers who could make useful contributions to improving our understanding, and from funders who could enable such research. Later, once the risks are better understood, there will be more need for attention from policy–makers, opinion leaders, and the general public to ensure that good countermeasures get implemented.

– What are your principal agreements and disagreements with Kurzweil's The Singularity Is Near?

I'm in agreement with much of what Kurzweil says. He has done a great job of informing large numbers of people about transhumanist ideas —in particular, prospects such as superintelligence, uploading, and advanced nanotechnology. That said, let me point to some possible differences of opinion:

I think there is more uncertainty about the time–scale for future technological developments, in particular superintelligence, than Kurzweil seems to think.

I don't think the exponential trends are quite as fast, robust, and reliable as Kurzweil believes. For many purposes, looking at world economic growth or productivity growth is a better measure of the overall rate of technological “progress” than looking at some particular technological fields where progress in recent years have happened to be particularly rapid.

I think it is more probable that the gap from roughly human–level artificial intelligence to radical superintelligence will be bridged much faster than Kurzweil seems to think. I am less confident that machine intelligence will be closely integrated with biological intelligence than Kurzweil seems to be.

I think Kurzweil is overoptimistic about the potential for present nutritional supplements to slow aging and greatly extend lifespan.

I am less confident than he seems to be that humans will avoid a terminal disaster from either nanotechnology or superintelligence.

I also think that some of the moral problems we are going to confront are deeper and more difficult than Kurzweil appears to believe. In particular, the problem of figuring out what kinds of lives would be most worth having if we abstract from current biological and technological limitations is not something one should be too casual about.

The characters in the dialogues he has in his book, even though some are supposedly superintelligences from the year 2100, are all remarkably human — indeed are drawn from a remarkably narrow corner of the space of present human personalities.

– Why do you believe superintelligence should be confronted before molecular nanotechnology?

Because they are quite likely to both happen roughly around the same time, and it seems that superintelligence would do more to protect against the dangers of nanotechnology than the other way around. If nanotechnology comes first, we would first have to survive the dangers of nanotechnology, then survive the dangers of superintelligence. If superintelligence comes first, we would still have to survive the dangers of superintelligence, but superintelligence might then guide us past the dangers of nanotechnology.

Eliezer Yudkowsky has also argued that nanotechnology may even increase the dangers of superintelligence by making available more computing power, which would lower the level of theoretical insight needed to create superintelligence, making it more likely that the first superintelligence will be created by people who don't have any deep understanding of what they are doing.

But these assessments are not set in stone. For example, someone who believed the safest route to superintelligence were some kind of coordinated upload–and–enhance scenario might favor a scenario where nanotechnology (as an enabler of uploading) comes first. To assess this, we would have to figure out the relative safety of attaining superintelligence through uploading versus other methods. We might also have to consider whether some method of uploading that does not rely on nanotechnology could feasibly be developed in the relevant time frame; and so forth. The issues soon become complicated when trying to think through our strategic options.

– You have written, “Although there is still a broad range of differing estimates that responsible thinkers could make, it is nonetheless arguable that because the negative utility of an existential disaster is so enormous, the objective of reducing existential risks should be a dominant consideration when acting out of concern for humankind as a whole.” You have also written, “For standard utilitarians, priority number one, two, three and four should consequently be to reduce existential risk. The utilitarian imperative 'Maximize expected aggregate utility!' can be simplified to the maxim 'Minimize existential risk!'”

Even when they agree with you, most people don't know what they can do to help minimize existential risk. What can Summit attendees do?

They could fund research on existential risk (either on such risks in general, or on particular risks such as those of AI or nanotech). They might also try to persuade others to provide such funding, or to generally raise awareness of the issues. Some attendees might also be in a position to do research themselves.

– Someone invests $5 million to further your goals. What do you do?

I would first discuss with the benefactor ideas for how to make optimum use of this donation, and what platform(s) would be best suited to implement the needed work. For a lot of research purposes, e.g. existential risks or ethical and practical issues related to human enhancement, or generally for big picture questions for humanity, the Oxford Future of Humanity Institute is uniquely well–placed for undertaking the appropriate research programs (by virtue of FHI's mission, of being part of a top research university, and of the people who are already in post). Other organizations also exist who work on specific angles —the Singularity Institute, obviously, working on friendly AI; also the Methuselah Prize, for rejuvenation research; the Institute for Ethics and Emerging Technologies, as a transhumanist think tank with potential future impact on the policy–wonk world; and the World Transhumanist Association, as a transhumanist grassroots–cultivating organization. Given the current paucity of funding, it would actually be possible with $5 million to substantially advance the work of all these organizations. In addition to the direct results that could be achieved with this investment, it would also — if allocated and used wisely — help to legitimate the whole general area of concern, which would have important follow–on effects such as attracting in more intelligent players to the field and stimulating further philanthropic interest.