The Singularity Institute Blog

By Dr. Ben Goertzel and Tyler Emerson

Why SIAI Exists

As the 21st century progresses, an increasing number of forward-thinking scientists and technologists are coming to the conclusion that this will be the century of AI: the century when human inventions exceed human beings in general intelligence. When exactly this will happen, no one knows for sure; Ray Kurzweil, for example, has estimated 2029.

Of course, where the future is concerned, nothing is certain except surprise; but the mere fact that so many knowledgeable people (such as Stephen Hawking, Douglas Hofstadter, Bill Joy, and Martin Rees) take the near advent of advanced AI as a plausible possibility, should serve as a “wake-up call” to anyone seriously concerned about the future of humanity.

Read the rest of this entry »

The third set of abstracts for the Singularity Summit, this September 8th and 9th at San Francisco’s beautiful Palace of Fine Arts. Tickets are only $50, and can be purchased here.

Innovative Applications of Early Stage AI
Neil Jacobstein, Teknowledge and Institute for Molecular Manufacturing

Early stage artificial intelligence has already produced a wide range of valuable but narrowly focused knowledge systems applications in industry and government. Many of these applications have performed complex tasks such as planning, monitoring, design, risk assessment, diagnosis, training, process control, classification, and analysis. For example, AAAI’s Innovative Applications of AI Conference has published hundreds of successful applications of AI. The applications are in fields as diverse as biotechnology, space flight, manufacturing, security, paleontology, construction, energy, music, military, intelligence, banking, telecommunications, news media, management, law, emergency services, agriculture, treaty verification, and many other areas. This talk will review the distribution of these applications across tasks and domains, and discuss the patterns that connect these applications: what worked, what didn’t, and what are the key trends. None of these systems exhibited general intelligence, but each documented our ability to codify and distribute human problem solving knowledge, and put it to work. The answer to the question about how far are we from advanced AI depends on the operational definition of “advanced”. It is clear from the knowledge systems produced thus far that even relatively straightforward applications can be valuable. The larger endeavor to produce AI systems that learn and reason at human levels and beyond is promising, and will require both enlightened research sponsorship and appropriate safeguards.

The Nature of Self-Improving Artificial Intelligence
Stephen M. Omohundro, Self-Aware Systems

Can we predict the behavior of systems that modify themselves? Can we design them to embody our values even after many generations of self-improvement? This talk will present a framework for answering questions like these. It shows that self-improving systems converge on a specific cognitive architecture that arose out of von Neumann’s foundational work on microeconomics. In these systems there is a universal principle which governs the organization of all levels of physical and computational resources. They exhibit four natural drives: 1) efficiency, 2) self-preservation, 3) resource acquisition, and 4) creativity. Unbridled, these lead to both desirable and undesirable behaviors.

The efficiency drive leads to algorithm optimization, data compression, atomically precise physical structures, reversible computation, adiabatic physical action, the virtualization of the physical, and governs a system’s choice of memories, theorems, language, and logic. The self-preservation drive leads to defensive strategies such as “energy encryption” for hiding resources and promotes replication and game theoretic modelling. The resource acquisition drive leads to a variety of competitive behaviors and promotes rapid physical expansion and imperialism. The creativity drive leads to the development of new concepts, algorithms, theorems, devices, and processes.

The best of these traits could usher in a new era of peace and prosperity; the worst are characteristic of human psychopaths and could bring widespread destruction. How can we ensure that this technology acts in alignment with our highest values? We have leverage both in designing the systems’ initial values and in creating the social context within which they operate. But we must have great clarity in imagining the future we want to create. We need not just a logical understanding of the technology but a deep introspection into what we cherish most. With both logic and inspiration we can work toward building a technology that empowers the human spirit rather than diminishing it.

Preparing for Bizarreness: Open Source Physical Security
Christine L. Peterson, Foresight Nanotech Institute

Attempting to take action now to get ready for a world with strong AI is a highly daunting task. Nevertheless it is worth considering our options, especially any that are useful in the nearer term for other reasons. We can ask: In a world of powerful entities, how can individuals be protected?

The open source software experience inspires us to look for ways to transfer the advantages of that process to the physical world. Open source has been particularly speedy at correction of security vulnerabilities – precisely the kind of vulnerabilities we will need to guard against in a world of highly powerful entities of various kinds. We can begin now to extend the principles of open source into the physical world: we can start to make physical security “bottom-up”, decentralized, collaborative, and transparent.

Download audio (mp3)

Cameron Reilly’s latest podcast interview is with Vernor Vinge, one of the best science fiction authors in recent decades. His novels include True Names, A Fire Upon the Deep, A Deepness in the Sky, and Rainbow’s End. He popularized the term “technological singularity” in his 1993 essay “The Coming Technological Singularity“.

At the Transvision conference, I had a conversation with a respected transhumanist on the issue of existential risks and humanity’s future. He told me that he did not see existential risk as a big deal because of it threatening hypothetical persons in the future, but only because of threatening the currently living population. This is the first time ever that anyone told me directly that they use a discount rate of infinity when considering as-yet-to-be-born persons.

When environmentalists tell us to fight against global warming, and economists warn us about the insolvency of Social Security, an often used motivator is to tell us to think about the world we are handing off to our children. This may refer to one’s own children, but can be abstracted to ‘descendants’ – which includes other people’s children, the sum total continuation of the human and eventually posthuman race.

What is confusing is that this motivator seems to work a lot on some people and not at all on certain others. Despite the majority assigning some level of concern to hypothetical persons, at least their immediate children, grandchildren, and even great-grandchildren, a significant minority assigns them nothing. Having no crystal ball of where moral philosophy and consensus will go in the future, it is very difficult for us to consider from our current vantage point whether or not this tendency will be viewed in retrospect as an irrational bit of evolutionary baggage, or genuine moral wisdom.

If we do care about hypothetical persons, and want to care more, might we eventually be able to reprogram our minds to magnify this aspect of ourselves as beneficial? Will we begin to care more about the teraperson hypothetical collective in the Whirlpool Galaxy, 25 million years from today, than our next-door neighbor we’ll see when we walk out the door in ten minutes? This certainly seems to be what Nick Bostrom is suggesting in his Astronomical Waste paper:

“With very advanced technology, a very large population of people living happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there is therefore an opportunity cost: a potential good, lives worth living, is not being realized. Given some plausible assumptions, this cost is extremely large. However, the lesson for utilitarians is not that we ought to maximize the pace of technological development, but rather that we ought to maximize its safety, i.e. the probability that colonization will eventually occur.”

Despite Bostrom’s persuasive paper, there are people that simply don’t care, because they only value lives as currently lived. As far as I can tell, there is nothing morally perverse about these people – they simply have a different angle on the moral issue. Is one “right” and the other “wrong”? Maybe, but ironically, it would require a hypothetical future person to give the answer with confidence.

Dr. Ben Goertzel is SIAI’s Director of Research. In this interview, he explains the Singularity Institute’s mission and research objectives. You can download the audio version here.

The second set of abstracts for the Singularity Summit, this September 8th and 9th at San Francisco’s beautiful Palace of Fine Arts. Tickets are only $50, and can be purchased here.

Metaverse Singularity
Jamais Cascio, Center for the Responsible Nanotechnology

There are numerous scenarios for how the Singularity might transpire, but implicit in most is the notion that the technologies that trigger the Singularity themselves emerge from earlier generations of systems and tools. One particularly rich potential progenitor is the spectrum of technologies encompassed by the term “Metaverse.” Building upon my work in the recently-published Metaverse Roadmap Overview, I trace how each of the four Metaverse scenarios – Augmented Reality, Lifelogging, Virtual Worlds and Mirror World – lead to very different types of Singularities. I look at the ways in which these different Singularity models might interact, and the implications each have for the likelihood of friendly and unfriendly AI.

Nine Years to a Positive Singularity – If We Really, Really Try
Dr. Ben Goertzel, Singularity Institute for Artificial Intelligence and Novamente

Common wisdom holds that powerful artificial general intelligence is decades to centuries off. Even techno-futurist Ray Kurzweil projects a date of 2029 for human-level AI via human brain emulation. My contention, however, is that powerful and beneficial AGI could come much sooner – if sufficient attention and resources are devoted to the right approaches. My favored approach involves integrating probabilistic and evolutionary learning, artificial economics, and other cutting-edge computer science techniques in a cognitive architecture informed by cognitive science and systems theory; and then embedding this architecture in virtual agents that interact with humans and each other in online virtual worlds. Among other advantages, I argue that this sort of AGI architecture is intrinsically better suited for stably ethical behavior than more closely human brain based architectures, due to the presence of a coherent and logical goal system. Current prototype work will be discussed, aimed at actualizing this approach via the release of intelligent agents controlled by the Novamente AI Engine in Second Life and other virtual worlds. Of course, it is difficult to place any kind of reliable estimate on the course of development of this kind of technology, given the R&D that remains to be done, and the uncertainties regarding funding and other practical exigencies. But radical success within less than a decade does not seem an outrageously unlikely possibility, in the view of this AGI researcher and entrepreneur.

Increased Intelligence, Improved Life
Peter Voss, Adaptive AI

Artificial General Intelligence (AGI) promises unprecedented advances not only in science and technology, but also in ethics and social systems. However, business – and thus consumers – will be first to experience some of the enormous benefits of this emerging technology. This talk will explore some of these improvements, and try to make a case for how increased intelligence leads to improved morality.

How much is an AI worth?

First, let’s distinguish between “instrumental” value and “intrinsic” value. Instrumental value is value to something else (usefulness); intrinsic value is inherent value that requires nothing else for it to be worth something. For example, we could say that clothing has instrumental value because they keep us warm, and our feeling of warmth has intrinsic value because it is worth something even in the absence of anything else. Or, we could say that books have instrumental value because they bring us knowledge and our having knowledge has inherent value. (For more, see value theory on Wikipedia.)

Read the rest of this entry »

Abstracts from Singularity Summit speakers are starting to arrive. Here are the first three.

The Singularity: A Period Not An Event
Rodney Brooks, MIT Computer Science and AI Laboratory

Whatever writes future history will look back at what we are calling the singularity not as a single event but as a period of time. The singularity period will encompass a time where a collection of technologies were invented, developed, and deployed in fits and starts, driven not by the imperative of the singularity itself, but by the normal economic and sociological pressures of human affairs. A Hollywood treatment of the singularity would have a world just like today’s, plus the singularity, as a singular event. In reality the world will be changing continuously due to rapid growth in technologies that are both related and unrelated to the singularity itself. The future will be embedded in a different world than the one we inhabit. And the AI systems we create will not have the same desires, beliefs, and goals as today-us. Tomorrow-us will be much better equipped for the changes that will take place in our world. This talk will explore how things might unfold and how we will transform ourselves along the way.

Waiting for the Great Leap…Forward?
Dr. James Hughes, Institute for Ethics and Emerging Technologies

Sentient, self-willed, greater-than-human machine minds are very likely in the next fifty years. But to ensure that they don’t threaten the welfare of the rest of the minds on the planet a number of steps need to be taken. First, given their radically different architecture and origins, developing software capacity for recognizing and relating to, perhaps having empathy for, human sentience should be a design goal, even if machine minds are likely to evolve beyond human perspectives and emotional traits. Second, building on the global networks established to identify and respond to computer viruses, governments and cyber-security firms need to develop detectors for and counter-measures for self-willed machine intelligence that may emerge, evolve, or be accidentally or maliciously released. Those detectors and counter-measures may or may not involve machine minds as well. Third, human beings should aggressively pursue cognitive enhancement and cyber-augmentation in order to give them a competitive chance against machine minds, economically and in th event of conflict. Fourth, since machine intelligence, self-willed or zombie, is likely to displace the need for most human occupations by the middle of the century, industrialized countries will need to renegotiate the relationship between education, work, income, and retirement, extracting a general social wage from robotic productivity to lift all boats, not just those of the shrinking group of workers and owners of capital. Finally, in order to ensure that we do not re-capitulate slavery, we will need to be much clearer about what kinds of minds, organic and machine, have what kinds of responsibilities and are owed which kinds of rights. Machine minds with a capacity to understand and obey the obligations of a democratic polity should be granted the rights to own property, vote and so on. Minds wishing to exercise capacities as dangerous as weapons or motor vehicles, should be licensed to do so, while even more dangerous capacities (AI equivalents of bombs) will need to be restricted to control by, or be integrated into the functioning of, accountable democratic governance.

The Road to Singularity: Comedic Complexity, Technological Thresholds, and Bioethical Broad Jumps
Wendell Wallach, Yale Interdisciplinary Center for Bioethics

The prospect of implementing higher order cognitive faculties in AI presumes that theories about the computational nature of mind are valid, that known technological issues can be solved, that there are no major surprise technological thresholds that will need to be crossed, and that computer scientists and public officials will find ways to navigate a broad array of ethical challenges. While some of these concerns have received considerable attention, others are just beginning to be noted. The ethical challenges, in particular, have not been well addressed. From robots carrying weapons, to moral decision making faculties for AI, to institutional review boards for robotic research, and political resistance to some categories of AI research, the bioethical challenges, if not addressed, could potentially undermine funding and public support for advanced AI systems. Progress in developing moral decision making faculties for computers is one area that engineers and designers can begin to tackle, and which will have a significant impact. The successful development of artificial moral agents (AMAs) is a major step that will help ameliorate other societal concerns regarding the development of advanced AI. The pathways for implementing moral decision making faculties in AI include top-down, bottom-up, and hybrid approaches. In addition, AMAs may require supra-rational faculties, such as social skills, emotions, consciousness, and a theory of mind.

Some skepticism about powerful AI and thus SIAI’s mission is reasonable, but it is important to have a sense of proportion. Even if you are skeptical about powerful AI to the point where you only give a 5% chance for its occurrence, it is important to balance your forecast with the expected utility residing within that 5% chance to realize an astronomically large social good. There are strong preexisting arguments in support of an astronomical amount of human value resulting from realizing powerful AI safely. You need to have wide confidence intervals in both directions given the current uncertainty residing within this new area of knowledge. You cannot just have it in the direction that favors your particular forecast. That 5% forecast also needs to account for the poor track record humans have as forecasters. Anyone who thinks their technical assessment of the issues is sound enough to justify a 5% (or 90%) forecast is either someone we should talk with, or someone showing sizable ignorance.

However, this misses the point: Even if it were possible to know that a 5% assessment were sound, the amount of social good that could be achieved by realizing powerful AI argues that the amount of effort presently marshaled in this direction is a rare form of human irrationality.

Unfortunately, very few people think like this. There are some remarkable people who already do, though, and I hope there are more soon. A few include Peter Thiel, Aubrey de Grey, and Barney Pell. I recommend exploring carefully how they think about these issues.

Consider the Stamp-Collecting Device. A common objection goes like this: “An optimization process that’s smart enough to tile the universe with stamps would also be smart enough to realize that this is not what its creator intended. Therefore it would not tile the universe with stamps.”

Human beings serve as a counterexample. The rules for constructing a human mind were devised by natural selection. These rules were fine-tuned to produce minds that are good at passing on their genes. If you are thinking of evolution as an optimization process, then it has the goal of producing genes which replicate as effectively as possible.

In 1859, Charles Darwin described the process that created us. Since then, we have come to understand that process in greater detail. Evolution is simple enough that we can claim to understand it very well; perhaps we even understand evolution as well as a Stamp-Collecting Device could understand us. Despite this understanding, we humans do not make evolution’s goal our own. Any time you use contraception, or perform a kind act when nobody is watching, you are betraying the goal of evolution. But so what? That’s evolution’s goal, not our goal. If anything, our understanding of evolution helps us to notice when we are doing something nasty but adaptive, and learn to avoid this behavior.

Similarly, a Stamp-Collecting Device would not adopt its programmer’s goals. It has its own goal to pursue — collecting stamps. If anything, understanding humans better would allow it to notice and fix biases that may be hindering its ability to collect stamps efficiently.

The challenge of FAI is to build an AI that does adopt our goals.