Research Grants

Academic Paper Grant

 
Existential Risk and Unknown Unknowns

Research summary:

Researchers have identified many potential existential disasters, e.g., nuclear war, bioterror, and unsafe artificial intelligence. It is unlikely, however, that all significant risks have been identified. This paper investigates what risks could be in this "everything else" category, how to assign probabilities to the category as a whole, and measures to reduced the expected harm of these unknown risks given our current state of knowledge.

Planned contents include:

  • A brief introduction to the idea of existential risks.
  • A section about what risks we know of now once used to be "everything else". How did they surprise us? What can we learn from them? 
  • A section discussing what categories we may find such risks in.
  • A section discussing the probability to assign to unknown risks, absolutely and relatively to known risks.
  • A section about policy implications: how can we deal with unknown unknowns?

Prior related work:

Existential Risks, by Nick Bostrom

Global Catastrophic Risks (edited by Nick Bostrom and Milan Cirkovic)

Target dates for
  • Extended Abstract (Posting an extended abstract on SIAI website, and circulating to related academics for comment): 2 weeks after start date.[1]
  • Working paper (Posting a working paper on the SIAI website; circulating to related academics): 6 weeks after start date.
  • Conference submission: 10 weeks after start date.
  • Follow-up steps (Brainstorming, and drafting proposals for, any follow-up publications. Should it be developed into a journal paper?): 12 weeks after start date.
[1] The "starting date" is the date (guaranteed to be within six months of the receipt of grant money) when we have skilled people to allocate to the project.  Extra donations increase our base of skilled people and thereby increase the number of projects we can get to; the lagged start date allows us to find new people, bring them here, and train them.

Total budget:  $4,400
Conference fees, air travel, motel: $1,400
Costs for researcher time: $3,000

How research costs are estimated:

  • Person-months for research and writing: 1.25 (This is our standard estimate of the time required for conference articles.[1])
  • Dollars required to support one skilled full time researcher-month[2]: $2,400

[1] Our base estimate is 1.25 person-months per conference paper, and 3 per journal article, for an experienced full-time researcher. This estimate takes the planning fallacy, and the importance of an outside view in avoiding that fallacy, into account. While typical rates of article production by professors are extremely low, the distribution is strongly skewed towards research-oriented universities and departments, and informal surveys of researchers working on existential risks give data consistent with this estimate for full-time work required per paper. Visiting Fellows vary in their experience levels, so that mean productivity is expected to be lower, but a team mix can be selected to account for this.

[2] This billing rate reflects an estimate of financial outlays for SIAI to create the equivalent of one full-time skilled researcher-month, including stipend or hosting expenses, workspace, and administrative or management time, and other supporting expenses. Actual person-months may be greater or lower depending on the labor mix for a particular project, with shortfalls made up from general funds. This rate is not reflective of the money researchers could earn in the competitive labor market. Think of this as a matched donation. You donate the living expenses; our researchers donate the surplus value of their labor.

 

How this paper will help reduce existential risk:

Research benefits  (What ideas will the paper explore?  How will that knowledge help with existential risk?):

  • The paper will explore existential risks and future technologies in general. The link to existential risk reduction is fairly direct: if we make it easier to find new risks, or if we identify ways to guard against any risk, we make it more likely that unforeseen kinds of disaster will be averted. The magnitude, timing, and predictability of other existential risks influence the expected positive and negative impacts of advanced AI.


Influence benefits (What target audience will the paper impact, how?  How will that impact help with existential risk?):

  • Drawing more attention to the category of "existential risks from causes that have not yet been identified" may cause others to undertake more efforts to combat such risks, to the extent possible.
  • The analytical framework is general and may attract cooperation from experts in risk analysis in general, but who do not share a particular interests in future technological risks.


Human capital benefits, or network benefits  (Will writing this paper help new Visiting Fellows become familiar with key research domains?  Will it help create relationships with outside co-authors?  Will it give folks interested in existential risk entry into new communities where valuable contacts may be found?):

  • Visiting Fellows working on this paper will get a better overview of the risk landscape. Collaboration with the Future of Humanity Institute is likely.



Credit card transactions are securely processed through PayPal.

PayPal – Grant Donation

(200 characters max)