• Nick Bostrom
• Cory Doctorow
• Max More

Interview

– What three important issues do we not understand well — what deserves greater attention? What forms do you think that “greater attention” should take?

Issue # 1: How to reliably elicit better reasoning, decision making, and creative thinking at all levels–international, governmental, organizational, and individual.

The great British leader, Winston Churchill once wrote: “We are shaping the world faster than we can change ourselves, and we are applying to the present the habits of the past.” His words were never more true or urgent. The accelerating pace of change and the growing complexity of the choices we face call for much smarter thinking–a combination and integration of convergent and divergent, critical and creative thinking, which we might call “integral intelligence.”

The human race faces a challenge that is unprecedented yet arises inevitably from our evolutionary history. This challenge comes with the advent of powerful new technologies capable of fundamentally reshaping human nature as well as the world around us. As we reach this historic threshold, we face both promise and peril. We are imperiling ourselves by failing to match scientific and technological advance with progress in decision making and envisioning.

We are racing into the future with outdated worldviews and unchallenged assumptions. We have started on a transhuman transition without the cognitive and institutional means for making sufficiently intelligent, well–informed, wise decisions about emerging technologies. If we don't get smarter as individuals, organizations, and societies, we will find ourselves on a collision course with a hazardous future. We face difficult, complex, and profound choices with brains that evolved in far simpler times. We find ourselves poorly equipped to understand and manage risk, uncertainty, and all but the most obvious and direct ramifications of changes in the complex systems of our world.

One part of this issue–an especially important part–is the need to come to a better understanding of the human need for certainty–and the conflict it produces. The destructive, inflammatory, controlling behaviors issuing from fundamentalist religion, nationalism, and other dogmas arise largely from the human need for certainty of belief and certainty of identity. We need to come to a better understanding of the biological, social, and neurological causes of our addiction to certainty. From an evolutionary perspective it no doubt has real value, so we will obviously have to move cautiously when ready to wean ourselves off it. But if we fail to do this, we cannot reasonably hope for a future devoid of ugly conflicts and suffering.

This issue folds several other issues within itself. Apart from the desire for certainty, another crucial aspect of the issue is how to overcome the “knowing-doing gap”. Some of us may know how to think more intelligently, but it doesn't follow that we will act on that knowledge. Even graduate students of statistics, for example, fall back on heuristics in cases when formal statistical rules would produce more accurate results.

Perhaps the knowing–doing gap will never be eliminated without the assistance of technology. So, another aspect of this issue is: How can we use technology to achieve “integral intelligence”? And, as an extension of this: How can we create intelligence at a level beyond human intelligence? This could take at least three major forms: augmentation of our own individual intelligence beyond current human limits; synthetic minds (AI); and distributed/social intelligence.

As to what forms this greater attention should take: Well, I'd say any forms that have a chance of working and that someone wants to tackle. For my part, I'll soon complete The Proactionary Principle, a book focusing on the challenges of thinking intelligently about a future that includes transhumanizing technologies.

Greater attention to this issue should include wider instruction in critical and creative thinking, starting earlier. (I taught critical thinking to college students for several years. This put me in a continual state of distress over the utter lack of instruction in reasoning and the effects of that neglect in my students.) I'd also like to see the development of social technologies to make “integral intelligence” easier and to motivate it. For the most part, the Web only enhances access to information, not to intelligence.

Finally, I see the need for a major program to encourage decision makers–especially at the policy level–to adopt the Proactionary Principle and related forms of structured decision making. This would mean taking an evidence–based approach to politics, economics, health, and risk management; and increasing the use of critical tools such as decision markets. 

Issue # 2. Responsibility and autonomy. The matter of what it is to be responsible and  autonomous has long been a central question of philosophy, but now belongs to a multidisciplinary realm including evolutionary biology, neuroscience, cognitive science, and AI. On an everyday level, this issue arises when we consider the difficulty of changing ourselves–as evidenced by the high levels of obesity, addiction, sedentary lifestyles, and numerous other unhealthy habits that individuals say they want to change, but don't.

In attempting to fully understand why we are responsible and autonomous to the extent that we are–and how we might enhance these quintessentially human qualities through the self-application of technology–we find ourselves wading through the muddy waters of human nature and personal identity. I think that, potentially, transhumanists could do much to illuminate this area. Progress in understanding will clearly be necessary as we develop the fields of neuroethics, “Friendly AI”, and the various approaches to self–programming and self–transformation.

Issue # 3. Aging. I don't need to say much about this. Involuntary aging and death is a rotten design feature for our species. It really is vital that we understand the causes of aging and how to intervene to stop it.

Greater attention should be given in the form of: Science prizes; increased funding; and heightened awareness among the public, especially among policy makers of the issues and possibilities. For instance, the current debates about the future of social security should take into account the potential effects of extended healthy lifespan–both what is likely from projecting trends, and what could be achieved if we had a consensus and a political will to combat aging.

– What are your principal agreements and disagreements with Kurzweil's The Singularity Is Near?

Agreements:
I'm very much on Ray Kurzweil's wavelength in terms of the law of accelerating returns. In fact, the first chapter of an earlier version of a book I was writing looked remarkably similar to Ray's first chapter in The Age of Spiritual Machines. I agree with Ray's observation that, as a corollary of the law of accelerating returns, we tend to overestimate change in the short term but underestimate it in the long run. In other words, our brains are attracted to the “intuitive linear view” rather than the “historical exponential view”. We find it difficult to fully think in terms of accelerating change because of the psychological phenomena of “anchoring” and “availability”. Our brains tend to fix on the recent rate of change because recent examples of the rate of change are most easily retrieved in memory, then use that as a basis for projecting future rates of change.

So I resonate strongly with the overall theme. I would diverge in that I'm not sure the acceleration over the long run is truly a law, or that it has always been followed precisely, and I'm not sure it's really a double exponential. Of course I'm also in agreement with Ray concerning the profound nature of the changes underway, as well with his emphasis on avoiding potential harms. I agree with Ray on so many specifics in The Singularity is Near (TSIN), that I'm not going to try to list them.

Disagreements:
I can think of three areas, as things stand right now, where I disagree with Ray's views as detailed in The Singularity is Near. I raise these points with some trepidation, because Ray might bury me at the bottom of that enormous store of historical data he's compiled. My disagreements—or doubts—concern the apparently automatic, inexorable nature of the technological trends; the rejection of the term “transhuman”; and the helpfulness of the term “Singularity.”

The first and most significant area where our views diverge concerns the nature of the forces behind accelerating change. As Ray presents them, these forces appear to be inexorable, almost automatic. Of course, he is not saying that progress will continue and accelerate if no one makes any effort to innovate. But he does seem to view the overall trend as a genuine metaphysical law, not merely a trend. Obviously, Ray does base his view of exponential acceleration into the Singularity on massive amounts of data. His book contains numerous fascinating charts showing trends that point in this direction, making especially evident the dramatic gains in computing power and drops in the energy needed for computing. He makes a strong (but well defended) claim that the rate of paradigm shift is accelerating, currently doubling every decade.

It's not just information technology that Ray sees as on an exponential highway into the future. The progress of nanotechnology means that, within a few decades, “the incremental cost of creating any physical product would be pennies per pound–basically the cost of raw materials.” The law of accelerating returns extends far beyond infotech and nanotech to encompass practically everything. Certainly, Ray intends it to include economic growth, even stating that the “law of accelerating returns is fundamentally an economic theory.”

Also he does say that the exponential trends driving productivity growth are only now entering the explosive growth phase, boldly claiming that the delay between early and late adoption will contract from about ten years today to just a couple of years by the mid-2020s (p.338). Not only will economic growth and productivity accelerate, bringing faster adoption with them, Ray assures us that “we'll see the underclass largely disappear over the next two or three decades.”

Now, I would like to buy into these claims, but I think they are only convincing from a purely technical perspective. Projections such as these are natural enough coming from Ray–the–computer scientist, Ray–the–technologist. But they become much harder to swallow whole if we take off our technologist hat and replace it with our economics, management, organizational psychology, political science, or cultural hats. I don't see anything to guarantee that we will automatically adopt new technologies at the pace specified by the law of accelerating returns. Even when we try to adopt new tech, it can take years to figure out how to use it effectively.

It's not clear to me how much we can compress that learning process–the process of integrating a technological innovation into our living and working practices. For instance, corporations have been using customer relationship management (CRM) software for quite a while, but a great many of them still haven't managed to reengineer their business processes, alter their incentives, and revise their business strategy and business model to enjoy the benefits. The same is true of many other technologies in business.

You can't simply plug a new technology into an organization (or a society) and expect optimal results to be produced reliably. You will have to figure out what changes to make in the organization's “DNA”, in its structure, decision rights, motivating factors, and information flows. Not all companies can readily adopt Cisco Systems' “Internet Culture” of distributed information flow, worker empowerment, and broad access to Web-based data for employees, suppliers, and customers. MIT's Erik Brynjolfsson has studied in depth the ways in which innovation in information technology needs to be matched by innovation in organizational practices. He has found, for instance, that “digital organizations” ease the shift of workers from functional specialists to process generalists, decentralize decision–making while supporting it, make use of effective incentive systems, coach employees in people skills, and give more autonomy and flexibility. These organizational changes take time, and the duration is directly tied to the underlying “substrate”, the people, who adjust to change at a steady rate, not an accelerating rate.

Other aspects of the argument for exponential economic effects of accelerating technology in TSIN give me pause. One chart depicts exponential software price–performance improvement. This support for exponential economic effects is highly selective however, focusing solely on speech recognition speed and accuracy. Another chart shows US patents granted since 1870, with a big jump since 1990. That looks impressive, but what about the possibility that patents are being granted more readily by overwhelmed examiners?

Economics may well need to be revised in light of technical developments. Still, I think Ray makes too sweeping a statement when he says that today's economic models are outdated because they depend on physical resources and the “intuitive linear” view, and that they ignore computation, knowledge, and intellectual property. Plenty of counterexamples exist, starting with Shapiro and Varian's book, Information Rules. I wouldn't dispute the overall picture portrayed in this respect by TSIN, but these points mean that I'm rather more reserved in my wild optimism.

The second area where we diverge–and the clearest of the three–is over the appropriateness of the terms “transhuman” and “posthuman.” I regard the condition we will be in post–Singularity as a posthuman condition. Transhumanists are those who anticipate the posthuman condition and see it as a desirable thing. In that sense, I would certainly count Ray as a transhumanist. However, Ray says (p.374) that the “whole idea of a species is a biological concept, and what we are doing is transcending biology.” He also says that “being human means being part of a civilization that seeks to extend its boundaries.” He remarks on the difficulty of drawing a line between a human and a posthuman–exactly how many million nanorobots do you need inside you before you cross the line into posthuman?

Now, this really isn't a big deal, but I continue to disagree over this. The fact that there is no sharp line dividing human from posthuman proves nothing, just as the fact of twilight doesn't prove that day never turns into night. I agree that the concept of our species is a biological concept, but that's precisely why I see us becoming posthuman. The edges of the species concept may be fuzzy, but the concept of our species is directly tied to our genetic structure and all that it enables and prevents. Once we have rendered our DNA essentially irrelevant, or at least vestigial, it will no longer make sense to refer to us as human. That won't make us inhuman–a concern that might crop up in response to the goal of going beyond human–just posthuman, or nonhuman, or whatever positively-defined species concept(s) we might then find useful.

The last of the areas of disagreement concerns the concept of the “Singularity”, although it's less a matter of disagreement than of emphasis. Ray does acknowledge potential dangers of Singularitarianism when it takes a passive form (p.373), so we're not far apart on this, but I remain somewhat leery of the concept because I think it reifies a continuing process. I'll say no more about this, however, since you can find a discussion on the first few pages of Chapter Seven in The Singularity Is Near. Ray and I carried out an extensive discussion of both this point and especially the first area above, in “Max More and Ray Kurzweil on the Singularity.” KurzweilAI.net, February 26, 2002.

– What subjective estimate would you make regarding the probability of an existential disaster within the next 10 years, and the next 20? Why?

First of all, I must say that I feel deeply uncomfortable answering this question. The only estimate I can give at this time is indeed a subjective one, and I'm not at all sure that has much value. I have no sound basis for making an estimate. I urge the reader—probably in vain–to remember this rather than the guesses that follow.

Within the next 10 years: 5% to  10%. Barring a large nuclear war, we won't have the capability to cause truly catastrophic disasters worthy of the label “existential”. The major risks during this near-term period are probably an outbreak of a mutated natural disease (such as a particularly virulent and transmissible form of avian flu), or a massive asteroid impact. The latter is more likely to destroy practically all human life, but the former is far more likely to happen in the next decade.

Within the next 20 years: Perhaps 15% to 20%. That's a pretty high estimate for such a dreadful outcome. I've based my guess on the suspicion that the years between one and two decades from now may well be the most dangerous we have ever faced or will face in the foreseeable future. There's always been a time lag between technological advance and our ability to handle it effectively. While that lag is gradually shrinking, advanced technologies may threaten drastically destructive outcomes for a significant time before we figure out how to control them. The actual probability of catastrophe depends on the relative rates of progress in potentially dangerous technologies and our institutional means of risk management, which themselves depend to some extent on technology. On the other hand, I'm not sure we face risks any greater than during the 1950s through the 1970s, when we were never very far from nuclear holocaust.

– Do you think an escalation of “singularity studies” would be valuable? If so, how would you recommend this escalation be sought?

Most descriptions of a technological Singularity foresee it being driven by the emergence of superintelligent machines, or “artificial” intelligence. In my view, real AI of human level or above will probably not come about as soon as projected by Ray Kurzweil and (perhaps) Eliezer Yudkowsky. The enormous implications of this development mean, however, that I fully support studies of how to bring it about and how to shape the event to minimize risks and maximize the outcomes desired by benevolent people. This would include studies of “Friendly AI”, which I think need to take a multidisciplinary approach.

I'd like to see Singularity studies that took the form of an organized and continuing effort to better foresee the implications of technological acceleration and cross–fertilization. My answer here essentially repeats my answer to the first question, concerning important issues that should receive more attention. From the perspective of what Singularity–relevant organizations should do, I'd say: more of the same, but even more smartly by committing to practicing what we might call evidence–based futurism.

– Someone offers you $5 million to further your goals. What do you do?

My answer here again ties into my answer to the first question. In brief, these are some of the things I would work with others to do:

• A major push to get the Proactionary Principle adopted nationally and internationally, to improve the way decisions are made about new technologies.
• Multidisciplinary classes in high schools and colleges, and seminars for businesses and other organizations, on technological change and more critical and creative thinking about its trajectories and effects.
• Bring together experts from diverse disciplines in a forum focused on the first issue in question 1, to develop better, evidence–based understanding of the problem. Develop practicable solutions for international organizations, governments, and government agencies, companies, and individuals.
• Create “action squads” of diverse experts to improve thinking and decision making in various areas. One example of this–though it seems to be a one–time approach, and makes recommendations only–is the Copenhagen Consensus.