The Singularity Institute Blog

“I think the Singularity Institute has some very smart people working on the most important mission on Earth, but… what exactly are they doing these days? I’m in the dark.”

There’s a good reason I hear this comment so often. We haven’t done a good job of communicating our progress to our supporters.

Since being appointed Executive Director of the Singularity Institute (SI) in November, I’ve been working to change that. I gave two Q&As about SI and explained our research program with a list of open problems in AI risk research. Now, I’d like to introduce our latest effort in transparency: monthly progress reports.

We begin with last month: December 2011. What did we do in December 2011?

(From this point on I’ll refer to myself as “Luke,” for clarity.)

• Winter fundraiser. We launched our winter fundraiser and have been contacting our supporters. The fundraiser has raised over $40k so far, though we still have $60k to go! (So, please donate!)
• Singularity Summit 2012. Our chief operating officer Amy Willey worked all month on preparations for Singularity Summit 2012, with much help from Luke. As a result we have now chosen a team of professionals with which we will take the Summit to “the next level,” and we’ve already confirmed several major speakers: Ray Kurzweil, Steven Pinker, Tyler Cowen, Temple Grandin, Peter Norvig, Robin Hanson, Peter Thiel, Melanie Mitchell, Vernor Vinge, and Carl Zimmer. We have also opened negotiations with many other speakers. This is a big improvement over our preparations for Singularity Summit 2011, which effectively began in May 2011, leaving us little time to capture certain speakers and develop certain kinds of media coverage. This much progress at such an early stage, in addition to a larger budget and greater professional assistance, will allow Singularity Summit 2012 to be a major leap forward for the event. Amy has also been developing arrangements for a possible European Singularity Summit in 2012.
• Rationality Org. As explained in our strategic plan, we recognize the branding confusion produced by focusing on both AI risk research and rationality education, so we are preparing to spin off a separate rationality education organization so that the Singularity Institute can focus on AI risk research. Internally, we are calling the rationality education organization “Rationality Org.” Anna and Eliezer, with some help from Luke, did a lot of work developing plans for the future Rationality Org. We spent even more time developing the core rationality lessons, testing versions of them on different groups of people, and iterating the content. We expect the Rationality Org to launch late this year or early next year, and we expect it to not only raise the sanity waterline but also bring significant funding toward existential risk reduction.
• New website design. Our media director Michael Anissimov, with much help from Luke, worked out the strategy and design of SI’s new website and worked with a designer to iterate the design several times. The designer is now programming the site.
• New donor database. In December, our Director of Development Louie Helm finished setting up our new donor database, including the custom code for automatically importing data from Paypal, Google Checkout, etc. This gives us a much better view of who our supporters are, and allows us to more effectively thank them for their support. Anissimov wrote personal thank-you notes to hundreds of past donors.
• Research articles. Luke and Anna made continued progress on their overview article “Intelligence Explosion: Evidence and Import.” Carl continued work with FHI‘s Stuart Armstrong on their article “Arms Races and Intelligence Explosions,” and continued work with Nick Bostrom on their article “How Hard is Artificial Intelligence? Evolutionary Arguments and Selection Effects.”
• Other articles. Luke wrote a few articles for Less Wrong: Hack Away at the Edges, Why study the cognitive science of concepts, and So You Want to Save the World. Eliezer made lots of progress on his new Bayes Theorem tutorial, including (outsourced) illustrations and much audience testing.
• Eliezer’s book. Eliezer finished the book proposal for his first book (already mostly written), The Science of Changing Your Mind. We have begun looking for good agents to represent the book.
• Facing the Singularity. Luke continued to develop his online book Facing the Singularity, a layman’s introduction to the Singularity, its consequences, and what we can do about it. The chapters he wrote in December 2011 were: The Crazy Robot’s Rebellion, Not Built to Think About AI, Playing Taboo with “Intelligence”, Superstition in Retreat, Plenty of Room Above Us, and Don’t Flinch Away.
• Additional transparency efforts. Anissimov and Luke began work on the design and content for an annual report. They also shot and produced Luke’s video Q&A #1.
• Optimal philanthropy. The optimal philanthropy movement (e.g. Giving What We Can) is growing exponentially. Carl and Anna did much collaboration and research with other members of the movement. Partly due to their work, the optimal philanthropy movement has great awareness of the case for existential risk reduction as optimal philanthropy, which should bring significant funding for existential risk reduction work in the coming years.
• Meetings with advisors, supporters, and potential researchers. During December 2011, various SI staff met or spoke with dozens of advisors, supporters, and collaborators about how to build the existential risk community, along with other topics. We also met with several potential researchers to gauge their interest and abilities.
• Google Adwords upgrade. For months, Louie and others have been tweaking the ads we get from $10k/month of Google Adwords donated to us by Google. By December 2011 our ads were so successful that we qualified for an upgrade, and are now receiving $40k/month of free advertising via Google Adwords.
• Better financial management. In December 2011 we began to train our new treasurer, long-time donor and friend of SI Jesse Liptrap. This means that someone outside the organization is keeping a close watch on our finances. We also began work on improving our book-keeping and accounting practices, which will allow better budgeting, forecasting, and resource management.
• New board member. Quixey co-founder and CEO Tomer Kagan was added to SI’s board of directors. Tomer is a good friend and brings a wealth of business and management experience to our team.
• Much more. Of course, we worked on dozens of other, smaller projects. These include: updates to IntelligenceExplosion.com; development of contacts for Rationality Org; the organization of regular SI staff dinners, to promote coordination and friendship; speaking with donors at Peter Thiel’s “Fast Forward” party; development of a database of helpful volunteers and assistants; implementing Olark on our donate page; meetings with reporters from various media organizations; uploading old videos to Vimeo and YouTube; fixing errors and outdated content on our website; finishing our 2010 990 and sent it to Brandon Reinhart to add to his financial examination of the Singularity Institute, preparing a new template for SI research publications (courtesy of research associate Daniel Dewey); and much more.

There is a new paper by Nick Bostrom and Eliezer Yudkowsky on the ethics of Artificial Intelligence. It will appear in the Cambridge Handbook of Artificial Intelligence:

The possibility of creating thinking machines raises a host of ethical issues.  These
questions relate both to ensuring that such machines do not harm humans and other
morally relevant beings, and to the moral status of the machines themselves.  The first
section discusses issues that may arise in the near future of AI.  The second section
outlines challenges for ensuring that AI operates safely as it approaches humans in its
intelligence.  The third section outlines how we might assess whether, and in what
circumstances, AIs themselves have moral status.  In the fourth section, we consider
how AIs might differ from humans in certain basic respects relevant to our ethical
assessment of them.  The final section addresses the issues of creating AIs more
intelligent than human, and ensuring that they use their advanced intelligence for
good rather than ill.

This paper serves as a good introduction to the problem of Friendly AI.

Singularity Institute Activities

Bugmaster asks:

…what does the SIAI actually do? You don’t submit your work to rigorous scrutiny by your peers in the field… you either aren’t doing any AGI research, or are keeping it so secret that no one knows about it… and you aren’t developing any practical applications of AI, either… So, what is it that you are actually working on, other than growing the SIAI itself ?

It’s a good question, and my own biggest concern right now. Donors would like to know: Where is the visible return on investment? How can I see that I’m buying existential risk reduction when I donate to the Singularity Institute?

SI has a problem, here, because it has done so much invisible work lately. Our researchers have done a ton of work that hasn’t been written up and published yet; Eliezer has been writing his rationality books that aren’t yet published; Anna and Eliezer have been developing a new rationality curriculum for the future “Rationality Org” that will be spun off from the Singularity Institute; Carl has been doing a lot of mostly invisible work in the optimal philanthropy community; and so on. I believe this is all valuable x-risk-reducing work, but of course not all of our supporters are willing to just take our word for it that we’re doing valuable work. Our supporters want to see tangible results, and all they see is the Singularity Summit, a few papers a year, some web pages and Less Wrong posts, and a couple rationality training camps. That’s good, but not good enough!

I agree with this concern, which is why I’m focused on doing things that happen to be both x-risk-reducing and visible.

First, we’ve been working on visible “meta” work that makes the Singularity Institute more transparent and effective in general: a strategic plan, a donor database (“visible” to donors in the form of thank-yous), a new website (forthcoming), and an annual report (forthcoming).

Second, we’re pushing to publish more research results this year. We have three chapters forthcoming in The Singularity Hypothesis, one chapter forthcoming in The Cambridge Handbook of Artificial Intelligence, one forthcoming article on the difficulty of AI, and several other articles and working papers we’re planning to publish in 2012. I’ve also begun writing the first comprehensive outline of open problems in Singularity research, so that interested researchers from around the world can participate in solving the world’s most important problems.

Third, there is visible rationality work forthcoming. One of Eliezer’s books is now being shopped to agents and publishers, and we’re field-testing different versions of rationality curriculum material for use in Less Wrong meetups and classes.

Fourth, we’re expanding the Singularity Summit brand, an important platform for spreading the memes of x-risk reduction and AI safety.

So my answer is to the question is: “Yes, visible return on investment has been a problem lately due to our choice of projects. Even before I was made Executive Director, it was one of my top concerns to help correct that situation, and this is still the case today.”

What if?

XiXiDu asks:

What would SI do if it became apparent that AGI is at most 10 years away?

This would be a serious problem because by default, AGI will be extremely destructive, and we don’t yet know how to make AGI not be destructive.

What would we do if we thought AGI was at most 10 years away?

This depends on whether it’s apparent to a wider public that AGI is at most 10 years away, or a conclusion based only on a nonpublic analysis.

If it becomes apparent to a wide variety of folks that AGI is close, then it should be much easier to get people and support for Friendly AI work, so a big intensification of effort would be a good move. If the analysis that AGI is 10 years away leads to hundreds of well-staffed and well-funded AGI research programs and a rich public literature, then trying to outrace the rest with a Friendly AI project becomes much harder. After an intensified Friendly AI effort, one could try to build up knowledge in Friendly AI theory and practice that could be applied (somewhat less effectively) to systems not designed from the ground up for Friendliness. This knowledge could then be distributed widely to increase the odds of a project pulling through, calling in real Friendliness experts, etc. But in general, a widespread belief that AGI is only 10 years away would be a much hairier situation than the one we’re in now.

But if the basis for thinking AI was 10 years away was nonpublic (but nonetheless persuasive to supporters who have lots of resources), then it could be used to differentially attract support to a Friendly AI project, hopefully without provoking dozens of AGI teams to intensify their efforts. So if we had a convincing case that AGI was only 10 years away, we might not publicize this but would instead make the case to individual supporters that we needed to immediately intensify our efforts toward a theory of Friendly AI in a way that only much greater funding can allow.

Budget

MileyCyrus asks:

What kind of budget would be required to solve the friendly AI problem?

Large research projects always come with large uncertainties concerning how difficult they will be, especially ones that require fundamental breakthroughs in mathematics and philosophy like Friendly AI does.

Even a small, 10-person team of top-level Friendly AI researchers taking academic-level salaries for a decade would require tens of millions of dollars. And even getting to the point where you can raise that kind of money requires a slow “ramping up” of researcher recruitment and output. We need enough money to attract the kinds of mathematicians who are also being recruited by hedge funds, Google, and the NSA, and have a funded “chair” for each of them such that they can be prepared to dedicate their careers to the problem. That part alone requires tens of millions of dollars for just a few researchers.

Other efforts like the Summit, Less Wrong, outreach work, and early publications cost money, and they work toward having the community and infrastructure required to start funding chairs for top-level mathematicians to be career Friendly AI researchers. This kind of work costs between $500,000 and $3 million per year, with more money per year of course producing more progress.

Predictions

Wix asks:

How much do members’ predictions of when the singularity will happen differ within the Singularity Institute?

I asked some Singularity Institute staff members to answer a slightly different question, one pulled from the Future of Humanity Institute’s 2011 machine intelligence survey:

Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of human-level machine intelligence? Feel free to answer ‘never’ if you believe such a milestone will never be reached.

In short, the survey participants’ median estimates (excepting 5 outliers) for 10%/50%/90% were:

2028 / 2050 / 2150

Here are five of the Singularity Institute’s staff members’ responses, names unattached, for the years by which they would assign a 10%/50%/90% chance of HLAI creation, conditioning on no global catastrophe halting scientific progress:

2025 / 2073 / 2168
2030 / 2060 / 2200
2027 / 2055 / 2160
2025 / 2045 / 2100
2040 / 2080 / 2200

Those are all the answers I had time to prepare in this round; I hope they are helpful!

Luke Muehlhauser: Robin, you’ve been donating $200 a month since August 2004. That adds up to more than $20,000, making you our 8th largest publicly listed donor! Why do you support the Singularity Institute like this?

Robin Powell: I honestly believe that a beneficial Singularity is the best hope that humanity has for long-term survival. Having spent hundreds of hours researching the various people and groups that are actively working on Singularity-related issues, the Singularity Institute is the only one that I really feel has their eyes on the right ball, which is the Friendly AI problem. I feel confident that my donations are the most effective way I can possibly aid in the best possible future for humanity.

Luke: What do you give up each month in order to donate $200/mo to the Singularity Institute?

Robin: Mostly I’ve been able to get by when things got complicated by re-budgeting, but I’ve had to do that rather a lot more often then I would have had to otherwise.

Luke: What challenges have you faced since August 2004, while continuing to donate $200 a month?

Robin: The time that I took off a couple of months to help my aging father, without pay, was by far the hardest; the extra money would really have helped then. But for me it’s about expected return: when the future of the human race is in the balance, having to borrow from friends briefly or similar hardships seem pretty inconsequential.

Luke: What one thought would you most like to share with the community of people who care about reducing existential risks?

Robin: AI is coming, relatively soon. There is no more important task for humanity than to prevent our extinction and preserve a better version of our values. Now is the time to spend time and money protecting the future of humanity. Please help us.

Luke: Thanks for your time, Robin, and thanks for your continued support!

The Singularity Institute and existential risk mitigation were recently covered in a special feature in Bloomberg Businessweek, and the article was syndicated on various websites, including SFGate.com. Bloomberg Businessweek has a circulation of approximately one million. A scan of the feature is available here.

From the New York Times Opinionator blog:

A robot walks into a bar and says, “I’ll have a screwdriver.” A bad joke, indeed. But even less funny if the robot says “Give me what’s in your cash register.”

The fictional theme of robots turning against humans is older than the word itself, which first appeared in the title of Karel Čapek’s 1920 play about artificial factory workers rising against their human overlords. Just 22 years later, Isaac Asimov invented the “Three Laws of Robotics” to serve as a hierarchical ethical code for the robots in his stories: first, never harm a human being through action or inaction; second, obey human orders; last, protect oneself. From the first story in which the laws appeared, Asimov explored their inherent contradictions. Great fiction, but unworkable theory.

Friendly AI is mentioned early on in the op-ed. The article makes the case why machine morality is important and why it is necessary to reconcile philosophical and engineering perspectives to make progress in this field.