Singularity Institute

Recent publications

• How Hard is Artificial Intelligence? The Evolutionary Argument and Observation Selection Effects (Carl Shulman and Nick Bostrom, 2012)
• Complex Value Systems are Required to Realize Valuable Futures (Eliezer Yudkowsky, 2011)
• Learning What to Value (Daniel Dewey, 2011)
• Ontological Crises in Artificial Agents' Value Systems (Peter de Blanc, 2011)
• Basic AI Drives and Catastophic Risks (Carl Shulman, 2010)
• Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics (Nick Tarleton, 2010)
• Economic Implications of Software Minds (S. Kaas, S. Rayhawk, A. Salamon and P. Salamon, 2010)
• From mostly harmless to civilization-threatening: pathways to dangerous artificial general intelligences (Kaj Sotala, 2010)
• Implications of a software-€limited singularity (Carl Shulman, Anders Sandberg, 2010)
• Superintelligence does not imply benevolence (Joshua Fox, Carl Shulman, 2010)
• Timeless Decision Theory (Eliezer Yudkowsky, 2010)
• Arms Control and Intelligence Explosions (Carl Shulman, 2009)
• Convergence of Expected Utility for Universal Artifical Intelligence (Peter de Blanc, 2009)
• Machine Ethics and Superintelligence (Carl Shulman, Henrik Jonsson, and Nick Tarleton, 2009)
• Which Consequentialism? Machine Ethics and Moral Divergence (Carl Shulman, Henrik Jonsson, and Nick Tarleton, 2009)

Prior Publications

• A Technical Explanation of Technical Explanation (Eliezer Yudkowsky, 2005)
• An Intuitive Explanation of Bayesian Reasoning (Eliezer Yudkowsky, 2003)
• Artificial Intelligence as a Positive and Negative Factor in Global Risk (Eliezer Yudkowsky, 2006)
• Artificial Intelligence as a Positive and Negative Factor in Global Risk [Chinese] (Eliezer Yudkowsky, 2006)
• Artificial Intelligence as a Positive and Negative Factor in Global Risk [Korean] (Eliezer Yudkowsky, 2006)
• Artificial Intelligence as a Positive and Negative Factor in Global Risk [Portuguese] (Elizer Yudkowsky, 2006)
• Cognitive Biases Potentially Affecting Judgment of Global Risks (Eliezer Yudkowsky, 2006)
• Coherent Extrapolated Volition (Eliezer Yudkowsky, 2004)
• Competent Program Evolution (Moshe Looks, 2006)
• Creating Friendly AI (Eliezer Yudkowsky, 2001)
• General Intelligence and Seed AI (Eliezer Yudkowsky, 2001)
• Levels of Organization in General Intelligence (Eliezer Yudkowsky, 2002)
• Mixing Cognitive Science Concepts with Computer Science Algorithms and Data Structures: An Integrative Approach to Strong AI (Moshe Looks and Ben Goertzel, 2006)
• Novamente: An Integrative Architecture for General Intelligence (Ben Goertzel et al., 2004)

Software

• The Uncertain Future is a web application that turns the user's subjective probability distributions on technological variables into output probabilities of AI and global disasters over time. It is currently in beta.
• Changing the frame of AI futurism: From storytelling to heavy-tailed, high-dimensional probability distributions (Steve Rayhawk et al., 2009) describes its motivation.

Talks and Working papers

• How intelligible is intelligence? (Anna Salamon, Stephen Rayhawk, JĂĄnos KramĂĄr, 2010)
• Whole Brain Emulation and the Evolution of Superorganisms (Carl Shulman, 2010)
• What can evolution tell us about the feasibility of artificial intelligence? (Carl Shulman, 2010)

Publications in Progress

• Eliezer Yudkowsky is writing a book on rationality, reductionism, and other prerequisites for thinking well about AI and Singularity outcomes. The book is targeted at bright students who may later contribute to Friendly AI research. Large parts of what will be the book are online in the form of sequences of posts at the rationality website Less Wrong.
• Eliezer Yudkowsky and other contributors to Less Wrong have been working on decision theory. They have been investigating frameworks by which agents with beliefs and goals may represent different possible actions and the consequences of those actions, and how the agents go from there to making decisions. This is especially relevant in the context of an AI fully capable of changing its own goal structure; knowing what decision theories are stable in such situations is one theoretical component needed for the construction of predictably safe AIs. Yudkowsky's "Timeless Decision Theory" is described in a few Less Wrong posts, as well as in a dedicated manuscript. Wei Dai and Vladimir Nesov have posted on "Updateless Decision Theory." Gary Drescher, the author of Good And Real, has also commented.