The Singularity Institute Blog

Singularity Institute Activities

Bugmaster asks:

…what does the SIAI actually do? You don’t submit your work to rigorous scrutiny by your peers in the field… you either aren’t doing any AGI research, or are keeping it so secret that no one knows about it… and you aren’t developing any practical applications of AI, either… So, what is it that you are actually working on, other than growing the SIAI itself ?

It’s a good question, and my own biggest concern right now. Donors would like to know: Where is the visible return on investment? How can I see that I’m buying existential risk reduction when I donate to the Singularity Institute?

SI has a problem, here, because it has done so much invisible work lately. Our researchers have done a ton of work that hasn’t been written up and published yet; Eliezer has been writing his rationality books that aren’t yet published; Anna and Eliezer have been developing a new rationality curriculum for the future “Rationality Org” that will be spun off from the Singularity Institute; Carl has been doing a lot of mostly invisible work in the optimal philanthropy community; and so on. I believe this is all valuable x-risk-reducing work, but of course not all of our supporters are willing to just take our word for it that we’re doing valuable work. Our supporters want to see tangible results, and all they see is the Singularity Summit, a few papers a year, some web pages and Less Wrong posts, and a couple rationality training camps. That’s good, but not good enough!

I agree with this concern, which is why I’m focused on doing things that happen to be both x-risk-reducing and visible.

First, we’ve been working on visible “meta” work that makes the Singularity Institute more transparent and effective in general: a strategic plan, a donor database (“visible” to donors in the form of thank-yous), a new website (forthcoming), and an annual report (forthcoming).

Second, we’re pushing to publish more research results this year. We have three chapters forthcoming in The Singularity Hypothesis, one chapter forthcoming in The Cambridge Handbook of Artificial Intelligence, one forthcoming article on the difficulty of AI, and several other articles and working papers we’re planning to publish in 2012. I’ve also begun writing the first comprehensive outline of open problems in Singularity research, so that interested researchers from around the world can participate in solving the world’s most important problems.

Third, there is visible rationality work forthcoming. One of Eliezer’s books is now being shopped to agents and publishers, and we’re field-testing different versions of rationality curriculum material for use in Less Wrong meetups and classes.

Fourth, we’re expanding the Singularity Summit brand, an important platform for spreading the memes of x-risk reduction and AI safety.

So my answer is to the question is: “Yes, visible return on investment has been a problem lately due to our choice of projects. Even before I was made Executive Director, it was one of my top concerns to help correct that situation, and this is still the case today.”

What if?

XiXiDu asks:

What would SI do if it became apparent that AGI is at most 10 years away?

This would be a serious problem because by default, AGI will be extremely destructive, and we don’t yet know how to make AGI not be destructive.

What would we do if we thought AGI was at most 10 years away?

This depends on whether it’s apparent to a wider public that AGI is at most 10 years away, or a conclusion based only on a nonpublic analysis.

If it becomes apparent to a wide variety of folks that AGI is close, then it should be much easier to get people and support for Friendly AI work, so a big intensification of effort would be a good move. If the analysis that AGI is 10 years away leads to hundreds of well-staffed and well-funded AGI research programs and a rich public literature, then trying to outrace the rest with a Friendly AI project becomes much harder. After an intensified Friendly AI effort, one could try to build up knowledge in Friendly AI theory and practice that could be applied (somewhat less effectively) to systems not designed from the ground up for Friendliness. This knowledge could then be distributed widely to increase the odds of a project pulling through, calling in real Friendliness experts, etc. But in general, a widespread belief that AGI is only 10 years away would be a much hairier situation than the one we’re in now.

But if the basis for thinking AI was 10 years away was nonpublic (but nonetheless persuasive to supporters who have lots of resources), then it could be used to differentially attract support to a Friendly AI project, hopefully without provoking dozens of AGI teams to intensify their efforts. So if we had a convincing case that AGI was only 10 years away, we might not publicize this but would instead make the case to individual supporters that we needed to immediately intensify our efforts toward a theory of Friendly AI in a way that only much greater funding can allow.

Budget

MileyCyrus asks:

What kind of budget would be required to solve the friendly AI problem?

Large research projects always come with large uncertainties concerning how difficult they will be, especially ones that require fundamental breakthroughs in mathematics and philosophy like Friendly AI does.

Even a small, 10-person team of top-level Friendly AI researchers taking academic-level salaries for a decade would require tens of millions of dollars. And even getting to the point where you can raise that kind of money requires a slow “ramping up” of researcher recruitment and output. We need enough money to attract the kinds of mathematicians who are also being recruited by hedge funds, Google, and the NSA, and have a funded “chair” for each of them such that they can be prepared to dedicate their careers to the problem. That part alone requires tens of millions of dollars for just a few researchers.

Other efforts like the Summit, Less Wrong, outreach work, and early publications cost money, and they work toward having the community and infrastructure required to start funding chairs for top-level mathematicians to be career Friendly AI researchers. This kind of work costs between $500,000 and $3 million per year, with more money per year of course producing more progress.

Predictions

Wix asks:

How much do members’ predictions of when the singularity will happen differ within the Singularity Institute?

I asked some Singularity Institute staff members to answer a slightly different question, one pulled from the Future of Humanity Institute’s 2011 machine intelligence survey:

Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of human-level machine intelligence? Feel free to answer ‘never’ if you believe such a milestone will never be reached.

In short, the survey participants’ median estimates (excepting 5 outliers) for 10%/50%/90% were:

2028 / 2050 / 2150

Here are five of the Singularity Institute’s staff members’ responses, names unattached, for the years by which they would assign a 10%/50%/90% chance of HLAI creation, conditioning on no global catastrophe halting scientific progress:

2025 / 2073 / 2168
2030 / 2060 / 2200
2027 / 2055 / 2160
2025 / 2045 / 2100
2040 / 2080 / 2200

Those are all the answers I had time to prepare in this round; I hope they are helpful!

Luke Muehlhauser: Robin, you’ve been donating $200 a month since August 2004. That adds up to more than $20,000, making you our 8th largest publicly listed donor! Why do you support the Singularity Institute like this?

Robin Powell: I honestly believe that a beneficial Singularity is the best hope that humanity has for long-term survival. Having spent hundreds of hours researching the various people and groups that are actively working on Singularity-related issues, the Singularity Institute is the only one that I really feel has their eyes on the right ball, which is the Friendly AI problem. I feel confident that my donations are the most effective way I can possibly aid in the best possible future for humanity.

Luke: What do you give up each month in order to donate $200/mo to the Singularity Institute?

Robin: Mostly I’ve been able to get by when things got complicated by re-budgeting, but I’ve had to do that rather a lot more often then I would have had to otherwise.

Luke: What challenges have you faced since August 2004, while continuing to donate $200 a month?

Robin: The time that I took off a couple of months to help my aging father, without pay, was by far the hardest; the extra money would really have helped then. But for me it’s about expected return: when the future of the human race is in the balance, having to borrow from friends briefly or similar hardships seem pretty inconsequential.

Luke: What one thought would you most like to share with the community of people who care about reducing existential risks?

Robin: AI is coming, relatively soon. There is no more important task for humanity than to prevent our extinction and preserve a better version of our values. Now is the time to spend time and money protecting the future of humanity. Please help us.

Luke: Thanks for your time, Robin, and thanks for your continued support!

The Singularity Institute and existential risk mitigation were recently covered in a special feature in Bloomberg Businessweek, and the article was syndicated on various websites, including SFGate.com. Bloomberg Businessweek has a circulation of approximately one million. A scan of the feature is available here.

From the New York Times Opinionator blog:

A robot walks into a bar and says, “I’ll have a screwdriver.” A bad joke, indeed. But even less funny if the robot says “Give me what’s in your cash register.”

The fictional theme of robots turning against humans is older than the word itself, which first appeared in the title of Karel Čapek’s 1920 play about artificial factory workers rising against their human overlords. Just 22 years later, Isaac Asimov invented the “Three Laws of Robotics” to serve as a hierarchical ethical code for the robots in his stories: first, never harm a human being through action or inaction; second, obey human orders; last, protect oneself. From the first story in which the laws appeared, Asimov explored their inherent contradictions. Great fiction, but unworkable theory.

Friendly AI is mentioned early on in the op-ed. The article makes the case why machine morality is important and why it is necessary to reconcile philosophical and engineering perspectives to make progress in this field.

Help us raise $100,000 that will go towards funding a website redesign, new research papers, and staging a bigger and better Singularity Summit 2012 in San Francisco. This fundraiser ends February 20th, so please contribute now to help us achieve our goal!

ARTIFICIAL INTELLIGENCE MORE RELEVANT THAN EVER

Recent books like Machine Ethics from Cambridge University Press and Robot Ethics from MIT Press, along with the U.S. military-funded research that resulted in Governing Lethal Behavior in Autonomous Robots show that the world is waking up to the challenges of building safe and ethical AI. But these projects focus on limited AI applications and fail to address the most important concern: how to ensure that smarter-than-human AI benefits humanity. The Singularity Institute has been working on that problem longer than anybody, a full decade before the Singularity landed on the cover of TIME magazine.

ACCOMPLISHMENTS IN 2011

2011 was our biggest year yet. Since the year began, we have:

• Held our annual Singularity Summit in New York City, with more than 900 in attendance. Speakers included inventor and futurist Ray Kurzweil, economist Tyler Cowen, PayPal co-founder Peter Thiel, Skeptic publisher Michael Shermer, Mathematica and WolframAlpha creator Stephen Wolfram, neuroscientist Christof Koch, MIT physicist Max Tegmark, and famed Jeopardy! contestant Ken Jennings.
• Held a smaller Singularity Summit in Salt Lake City.
• Held a one-week Rationality Minicamp and a ten-week Rationality Boot Camp.
• Created the Research Associates program, which currently has 7 researchers coordinating with Singularity Institute.
• Published our Singularity FAQ, IntelligenceExplosion.com, and Friendly-AI.com.
• Wrote three chapters for Springer’s upcoming volume The Singularity Hypothesis, along with four other research papers.
• Began work on a new, clearer website design with lots of new content, which should go live Q1 2012.
• Began outlining open problems in Singularity research to help outside collaborators better understand our research priorities.

FUTURE PLANS YOU CAN HELP SUPPORT

In the coming year, we plan to do the following:

• Hold our annual Singularity Summit, in San Francisco this year.
• Improve organizational transparency by creating a simpler, easier-to-use website that includes Singularity Institute planning and policy documents.
• Publish a document of open research problems in Singularity Research, to clarify the research space and encourage other researchers to contribute to our mission.
• Add additional skilled researchers to our Research Associates program.
• Publish a well-researched document making the case for existential risk reduction as optimal philanthropy.
• Diversify our funding sources by applying for targeted grants and advertising our affinity credit card program.

We appreciate your support for our high-impact work.

Follow the progress of our fundraiser at http://singinst.org/2011winterfundraiser.

The Singularity Summit was a huge success. We raised over $300,000, one-third of that a donation from Jaan Tallinn, another third from a variety of generous donors who gave during or immediately after the Summit, and a third from ticket sales. We have roughly $500,000 in the bank, while annual payroll expenses are about $350,000. We almost have enough money to make it through the next year without adding any research staff, though we definitely would like to add additional researchers. In the meantime, our (unpaid) Research Associates team has been growing, and is tackling a variety of projects.

Since the Singularity Summit in October, our President Michael Vassar moved on to help found Personalized Medicine, a company we are all excited about, and Luke Muehlhauser was appointed Executive Director. Luke answered questions about these changes and our future plans in a recent video Q&A. I encourage you to watch it, and submit additional questions you may have about the Singularity Institute.

We have seven staff at this time: Luke, Michael Anissimov, Anna Salamon, Carl Shulman, Amy Willey, Louie Helm, and Eliezer Yudkowsky. Our internal collaboration has increased, we’re keeping work logs, and we regularly eat dinner together. It’s becoming more of a family.

What are we all doing? Luke and Michael are working on a new website for the Singularity Institute. Amy and Luke are working hard to expand the Singularity Summit brand and bring the Summit to “the next level.” Anna and Eliezer are building a curriculum for the new Rationality Org, which we hope to spin off as a separate organization from Singularity Institute sometime next year. Luke, Anna, and Carl have been working on a variety of research papers, and Carl has also been working with the fast-growing “optimal philanthropy” movement, which is now poised to direct substantial funds over the next few years to reducing existential risks. Louie is working on donor relations, fundraising, recruiting, operations, and has also contributed to some of our forthcoming research articles.

In the coming year, we look forward to improving communication and transparency with our supporters, and to increasing the rate of our published research output.

A full transcript of this video is hosted at Less Wrong.

Louie Helm and Luke Muehlhauser, Singularity Institute staff, are sharing a draft of their chapter submission to The Singularity Hypothesis edited volume for feedback:

The Singularity and Machine Ethics

Abstract: Many researchers have argued that a self-improving artificial intelligence (AI) could become so vastly more powerful than humans that we would not be able to stop it from achieving its goals. If so, and if the AI’s goals differ from ours, then this could be disastrous for humans. One proposed solution is to program the AI’s goal system to want what we want before the AI self-improves beyond our capacity to control it. Unfortunately, it is difficult to specify what we want. After a brief digression concerning human intuitions about intelligence, we offer a series of “intuition pumps” in moral philosophy for our conclusion that human values are complex and difficult to specify. We then survey the evidence from the psychology of motivation, moral psychology, and neuroeconomics that supports our position. We conclude by recommending ideal preference theories of value as a promising approach for developing a machine ethics suitable for navigating the Singularity.

Here is an index of all Singularity Summit speeches.

Summit 2006

• Todd Davies, Tyler Emerson, and Peter Thiel. Introduction to the Singularity Summit.
• Ray Kurzweil. The Singularity: a hard or soft takeoff?
• Douglas Hofstadter. Trying to muse rationally about the Singularity scenario.
• Nick Bostrom. Artificial Intelligence and existential risks.
• Sebastian Thrun. Toward human-level intelligence in autonomous cars.
• Cory Doctorow. Singularity or Dark Age?.
• K. Eric Drexler. Productive Nanosystems: Toward a Super-Exponential Threshold in Physical Technology.
• Max More. Cognitive and Emotional Singularities: Will Superintelligence come with Superwisdom?
• Christine Peterson. Bringing Humanity and the Biosphere through the Singularity
• John Smart. Searching for the Big Picture: Systems Theories of Accelerating Change
• Eliezer Yudkowsky. The Human Importance of the Intelligence Explosion.
• Bill McKibben. Being good enough.
• Ray Kurzweil. Follow up.
• Panel Discussion and Q&A.
• Panel Discussion and Q&A pt. 2.

Summit 2007

• Tyler Emerson & Peter Thiel. Welcome and introduction.
• Rodney Brooks. The Singularity: A Period Not An Event.
• Eliezer Yudkowsky. Introducing the “Singularity”: Three Major Schools of Thought.
• Barney Pell. Pathways to Advanced General Intelligence: Architecture, Development, and Funding.
• Wendell Wallach. Superstition and Forgetfulness – Two Essentials for Artificial General Intelligence.
• Barney Pell, Wendell Wallach, Sam Adams. First panel discussion.
• Jamais Cascio. Metaverse Singularity.
• Stephen M. Omohundro. The Nature of Self-Improving Artificial Intelligence.
• Peter Voss. Improved intelligence, improved life.
• Stephen M. Omohundro, Peter Voss. Second panel discussion.
• Neil Jacobstein. Innovative Applications of Early Stage AI.
• Ben Goertzel. Nine Years to a Positive Singularity – If We Really, Really Try.
• Paul Saffo. Machines of Loving Grace: Anticipating Advanced AI.
• Neil Jacobstein, Ben Goertzel, Paul Saffo. Third panel discussion.
• Peter Norvig. The history and future of technological change.
• J. Storrs Hall. Asimov’s laws of robotics — revised.
• Peter Thiel. Financial markets and the Singularity.
• Charles L. Harper, Jr. Superintelligence, the “Dilemma of Power,” and the transformation of desire.
• J. Storrs Hall, Peter Thiel, Charles L. Harper, Jr. Third panel discussion.
• Steve Jurvetson. Dichotomy of designed and evolutionary paths to AI futures.
• Christine L. Peterson. Preparing for bizarreness: open source physical security.
• James Hughes. Waiting for the Great Leap…Forward?
• Eliezer Yudkowsky. The Challenge of Friendly AI.
• Christine L. Peterson, James Hughes, Eliezer Yudkowsky. A dialogue with Ray Kurzweil.

Summit 2008

• Vernor Vinge and Bob Pisani. Conversation on the Singularity.
• Esther Dyson. 23andme and personal genomics.
• James Miller. Societal reactions to the Singularity.
• Eric Baum. AI and the problem of understanding.
• Dharmendra Modha. IBM’s research into Whole Brain Emulation.
• Ben Goertzel. OpenCog — an open source AGI project.
• Marshall Brain. Robotics and structural unemployment.
• Cynthia Breazeal. Social robots.
• Ray Kurzweil, Glen Zorpette and John Horgan. Debate on the Singularity.
• Pete Estep. The InnerSpace Foundation.
• Neil Gershenfeld. Alternate models of computing.
• Peter Diamandis. History of the X Prize Foundation and future X Prizes.
• Ray Kurzweil. Exponential progress in information technologies.
• Justin Rattner. Intel and the continuous of Moore’s law.
• Nova Spivack. Collective intelligence and the emerging global brain.

Summit 2009

• Michael Vassar. Introduction.
• Anna Salamon. Shaping the intelligence explosion.
• Anders Sandberg. Technical roadmap for whole brain emulation.
• Randal Koene. The time is now: as a species we need whole brain emulation.
• Itamar Arel. Technological convergence leading to artificial general intelligence.
• Ben Goertzel. Pathways to beneficial artificial general intelligence.
• Stuart Hameroff. Neural substrates of consciousness and the ‘conscious pilot’ model.
• David Chalmers. Simulation and the singularity.
• Gary Drescher. Choice machines, causality, and cooperation.
• Ed Boyden. Synthetic neurobiology: optically engineering the brain to augment its function.
• Marcus Hutter. Foundations of intelligent agents.
• William Dickens. Cognitive ability: past and future enhancements and implications.
• Ray Kurzweil. The ubiquity and predictability of the exponential growth of information technology.
• Bela Nagy. More than Moore: comparing forecasts of technological progress.
• Robin Hanson. How does society identify experts, and when does it work?
• Panel: Future of scientific method.
• Gregory Benford. Artificial biological selection for longevity.
• Ray Kurzweil. Critics of the singularity.
• Brad Templeton. The finger of AI: Automated electrical vehicles and oil independence.
• Gary Marcus. The fallibility and improvability of the human mind.
• Peter Thiel. Macroeconomics and singularity.
• Aubrey de Grey. The Singularity and the Methuselarity: similarities and differences.
• Thiel, Yudkowsky & de Grey panel: Changing the world.
• Anna Salamon. How much it matters to know what matters: A back of the envelope calculation.
• Gary Wolf. The petaflop macroscope.
• Eliezer Yudkowsky. Cognitive biases and giant risks.
• Jurgen Schmidhuber. Compression progress: The algorithmic principle behind curiosity and creativity.
• Thiel, Rose & Gorenberg Panel: Venture capitalism.
• Michael Nielsen. Quantum computing: What it is, what it is not, what we have yet to learn.
• Michael Nielsen. Collaborative networks in scientific discovery.
• Stephen Wolfram. Conversation on the singularity.

Summit 2010

• Michael Vassar. The Darwinian method.
• Gregory Stock. Evolution of post-human intelligence.
• Ray Kurzweil. The mind and how to build one.
• Ben Goertzel. AI against aging.
• Steven Mann. Humanistic intelligence augmentation and mediation.
• Mandayam Srinivasan. Enhancing our bodies and evolving our brains.
• Brian Litt. The past, present and future of brain machine interfaces.
• Demis Hassabis. Combining systems neuroscience and machine learning: a new approach to AGI.
• Terry Sejnowski. Reverse-engineering brains is within reach.
• Dennis Bray. What cells can do that robots can’t.
• Terry Sejnowski/Dennis Bray debate: Simplified humanism and positive futurism.
• Ramez Naam. The digital biome.
• Lance Becker. Modifying the boundary between life and death.
• Ellen Heber-Katz. The MRL mouse – how it regenerates and how we might do the same.
• Shane Legg. Universal measures of intelligence.
• John Tooby. Can discovering the design principles governing natural intelligence unleash breakthroughs in AI?.
• Tooby, Goertzel, Yudkowsky & Legg panel. Narrow and General Intelligence.
• David Hanson. David Hanson: Why Characters Are Key to Friendly A.I..
• Irene Pepperberg. Irene Pepperberg: Nonhuman Intelligence: Where we are and where we’re headed.
• James Randi. Is there such a thing as scientific consensus?

Summit 2011

• Ray Kurzweil. From Eliza to Watson to passing the Turing Test.
• Stephen Badylak. Regenerative medicine: possibilities and potential.
• Sonia Arrison. 100 Plus: how the coming age of longevity will change everything, from careers and relationships to family and faith.
• Peter Thiel. Back to the future.
• James McLurkin. The future of robotics is swarms: why a thousand robots are better than one.
• Michael Shermer. Social Singularity: transitioning from civilization 1.0 to 2.0.
• Jason Silva. The ‘Undivided Mind’ — science and imagination.
• Stephen Wolfram. Computation and the future of mankind.
• Dmitry Itskov. Project ‘Immortality 2045′ — Russian experience.
• Christof Koch. The neurobiology and mathematics of consciousness.
• Eliezer Yudkowsky. Open problems in friendly artificial intelligence.
• Max Tegmark. The future of life: a cosmic perspective.
• Alexander Wissner-Gross. Planetary-scale intelligence.
• Sharon Bertsch McGrayne. A History of Bayes theorem.
• David Brin. So you want to make gods. Now why would that bother anybody?
• Tyler Cowen. The Great Stagnation.
• Tyler Cowen & Michael Vassar. Debate on the Great Stagnation.
• John Mauldin. The endgame meets The millennium wave — why the economic crisis will be history as we create the future.
• Riley Crane. Rethinking communication.
• Dileep George and Scott Brown. From planes to brains: building AI the Wright way.
• Jaan Tallinn. Balancing the trichotomy: individual vs. society vs. universe.
• David Ferrucci. Watson AI perceptions.
• Dan Cerutti. Commercializing Watson.
• Ken Jennings. The human brain in Jeopardy: computers that ‘think’.

See them all at our YouTube channel.

More than 500 photos are also online at our Flickr account.