Singularity Institute

[1] The "starting date" is the date (guaranteed to be within six months of the receipt of grant money) when we have skilled people to allocate to the project.  Extra donations increase our base of skilled people and thereby increase the number of projects we can get to; the lagged start date allows us to find new people, bring them here, and train them.

Total budget:  $5,960, including:

• Conference fees, air travel, motel: $1,400

• Costs for researcher time: $4,560

How research costs are estimated:

• Person-months for research and writing: 1.9 (obtained by taking our standard estimate[1] of 1.25 person-months per conference paper and multiplying by 1.5, since this paper requires thinking through and aggregating many different topics).
• Dollars required to support one skilled full time researcher-month[2]: $2,400

[1] Our base estimate is 1.25 person-months per conference paper, and 3 per journal article, for an experienced full-time researcher. This estimate takes the planning fallacy, and the importance of an outside view in avoiding that fallacy, into account. While typical rates of article production by professors are extremely low, the distribution is strongly skewed towards research-oriented universities and departments, and informal surveys of researchers working on existential risks give data consistent with this estimate for full-time work required per paper. Visiting Fellows vary in their experience levels, so that mean productivity is expected to be lower, but a team mix can be selected to account for this.  Ramsden and Fox report similarly low results.

[2] This billing rate reflects an estimate of financial outlays for SIAI to create the equivalent of one full-time skilled researcher-month, including stipend or hosting expenses, workspace, and administrative or management time, and other supporting expenses. Actual person-months may be greater or lower depending on the labor mix for a particular project, with shortfalls made up from general funds. This rate is not reflective of the money researchers could earn in the competitive labor market. Think of this as a matched donation. You donate the living expenses; our researchers donate the surplus value of their labor.
 

How this paper will reduce existential risk:

Research benefits (What ideas will the paper explore?  How will that knowledge help reduce existential risk?)

• The paper will explore ways in which decision theory and anthropic reasoning play into our decisions around existential risk. It can thereby both: (a) improve our picture of the existential risks landscape, and (b) improve our knowledge of what future research along these lines can or cannot be expected to offer good returns in existential risk reduction.

Influence benefits(What target audience will the paper impact, how?  How will that impact help with existential risk?)

• To the extent that the behavior of AIs critically depends on the decision theory or on the solution to anthropic puzzles that their decision algorithms approximate, this identifies an additional key failure mode for the creation of Friendly AI, alongside recognized possibilities of skewed Bayesian priors and inhumane utility functions; this helps to illustrate the difficulty of creating safe and beneficial powerful AI and the risk of failure.

• The paper can help SIAI and FHI researchers, as well as others who care about existential risk, know how much research effort to put into decision theory, simulation probabilities, and anthropics.

• The paper may also increase the number of mathematically and philosophically talented thinkers who engage with AI risk, insofar as such researchers are attracted to these puzzles.
• Insofar as the paper sheds direct light on decision theory, anthropics, and related topics, and insofar as these issues do in fact bear on existential risk reduction strategies, the paper may influence us to use better strategies.

Human capital benefits, or network benefits (Will writing this paper help new Visiting Fellows become familiar with key research domains?  Will it help create relationships with outside co-authors?  Will it give folks interested in existential risk entry into new communities where valuable contacts may be found?)

• This paper will help Visiting Fellows become familiar with the problems of decision theory and observer selection effects in philosophy. It may forge connections with capable philosophers, economists, and others in those fields and pave the way for collaborations on other topics of joint interest.