Research Grants

Academic Paper Grant


Why AI Risk Demands Better Scientific Methodologies


Research summary:
Successfully navigating AI risk demands new predictive methodologies. In particular, it demands that scientists: (a) successfully discern which technologies would lead to existential catastrophes, before having any concrete experience of such catastrophes; and (b) convince relevant decision-makers that such risks are real.

This paper will consider what methodological improvements, if any, might increase our odds of achieving (a) and (b). It will survey existing literature on methods that can help groups make accurate predictions in contexts with sparse data and strong potential for distorting emotions. It will estimate the extent to which improvements in decision-making methodologies or institutions can expected to reduce AI risks, as well as identify some of the specific types of research into methodological improvements that most warrant further investigation.

Prior related work:


Venues for presentation:

A conference on risk analysis, on scientific methodology and practice (e.g., a philosophy of science conference), or on alternative institutions.


Target dates for:

Extended abstract (Posting an extended abstract on SIAI website, and circulating to related academics for comment): 3 weeks after start date.[1]

 

Working paper (Posting a working paper on the SIAI website; circulating to related academics): 9 weeks after start date.

 

Conference submission: 15 weeks after start date.

 

Follow-up steps (Brainstorming, and drafting proposals for, any follow-up publications. Should it be developed into a journal paper?): 18 weeks after start date.  

[1] The "starting date" is the date (guaranteed to be within six months of the receipt of grant money) when we have skilled people to allocate to the project.  Extra donations increase our base of skilled people and thereby increase the number of projects we can get to; the lagged start date allows us to find new people, bring them here, and train them.

Total budget:  $5,900
  • Conference fees, air travel, motel: $1,400
  • Costs for researcher time: $4,500

How research costs are estimated:
  • Person-months for research and writing: 1.875 (Our standard estimate for conference papers[1] times 1.5, because this paper requires integrating knowledge of several different subproblems around decision theory and anthropics and so can be expected to take longer than average.)
  • Dollars required to support one skilled full time researcher-month[2]: $2,400

[1] Our base estimate is 1.25 person-months per conference paper, and 3 per journal article, for an experienced full-time researcher. This estimate takes the planning fallacy, and the importance of an outside view in avoiding that fallacy, into account. While typical rates of article production by professors are extremely low, the distribution is strongly skewed towards research-oriented universities and departments, and informal surveys of researchers working on existential risks give data consistent with this estimate for full-time work required per paper. Visiting Fellows vary in their experience levels, so that mean productivity is expected to be lower, but a team mix can be selected to account for this.

[2] This billing rate reflects an estimate of financial outlays for SIAI to create the equivalent of one full-time skilled researcher-month, including stipend or hosting expenses, workspace, and administrative or management time, and other supporting expenses. Actual person-months may be greater or lower depending on the labor mix for a particular project, with shortfalls made up from general funds. This rate is not reflective of the money researchers could earn in the competitive labor market. Think of this as a matched donation. You donate the living expenses; our researchers donate the surplus value of their labor.


How this paper will help reduce existential risk:


Research benefits (What ideas will the paper explore?  How will that knowledge help reduce existential risk?):
  • The proposed research will sharpen our ideas on what kinds of institutions will enable humanity to safely navigate future technologies.
  • It can also help us estimate the returns (in existential risk reduction) from future research into better predictive institutions.

Influence benefits (What target audience will the paper impact, how?  How will that impact help with existential risk?):
  • If institutional improvements are a promising angle toward existential risk reduction, this paper may:
    • Increase interest in the most useful known predictive institutions;
    • Increase research toward future institutional improvements.


Human capital benefits, or network benefits (Will writing this paper help new visiting fellows become familiar with key research domains?  Will it help create relationships with outside co-authors?  Will it give folks interested in existential risk entry into new communities where valuable contacts may be found?):

  • Institutional improvements may well be a major leverage point for affecting global outcomes, including outcomes around AI risk.  SIAI has had little contact with such academics to date, with the exception of Robin Hanson; this paper would give us an opportunity to increase contact with this research community.


Credit card transactions are securely processed through PayPal.

PayPal – Grant Donation

(200 characters max)