Research Grants
Academic Paper Grant
Machine Ethics and Superintelligence
Research summary:
The developing field of machine ethics, galvanized by technologies such as military robotics, is a burgeoning community that often conducts general analysis on the problem of creating Artificial Moral Agents (AMAs) of varying capabilities, and does not only focus on near-term developments. This paper will analyze the special demands of superintelligent machines for machine ethics, assumptions that can generally be made safely with respect to less capable systems but do not hold with respect to superintelligence.
Planned contents include:
- Background on the ideas of superintelligence and Singularity or intelligence explosions.
- Existential risk potential of superintelligence as a reason to analyze the matter in advance.
- The need for reflective consistency in superintelligent systems.
- The more limited applicability of reward and punishment to superintelligent systems.
- The difficulty of producing testing environments for superintelligent systems that capture all relevant features.
- Challenges facing human attempts to closely monitor incremental improvements to AMAs as capabilities approach superintelligence.
Prior related work:
- An SIAI conference presentation at the Asia-Pacific Computing and Philosophy conference
- Eliezer Yudkowsky's book chapter "Artificial Intelligence as a Positive and Negative Factor in Global Risk" (Global Catastrophic Risks, Nick Bostrom & Milan Cirkovic (eds.) Oxford University Press, 2007)
Venues for presentation or publication:
The Minds and Machines special issue on "transhumanism, cognitive enhancement, and AI" (submission deadline January 15).
How research costs are estimated:
- Person-months for research and writing: 3 (This is our standard estimate of the time required for journal articles[1].)
- Dollars required to support one skilled full time researcher-month[2]: $2,400
[2] This billing rate reflects an estimate of financial outlays for SIAI to create the equivalent of one full-time skilled researcher-month, including stipend or hosting expenses, workspace, and administrative or management time, and other supporting expenses. Actual person-months may be greater or lower depending on the labor mix for a particular project, with shortfalls made up from general funds. This rate is not reflective of the money researchers could earn in the competitive labor market. Think of this as a matched donation. You donate the living expenses; our researchers donate the surplus value of their labor.
Target dates for:
Follow-up steps (Brainstorming, and drafting proposals for, any follow-up publications. Are there related research papers that should be considered?): Jan 30, 2010.
How this paper will help reduce existential risk:
Research benefits (What ideas will the paper explore? How will that knowledge help with existential risk?):
- Research benefits will be minimal, as the paper is primarily aimed at clear exposition and review of existing work.
Influence benefits (What target audience will the paper impact, how? How will that impact help with existential risk?):
- The paper will be aimed at the machine ethics community in hopes of inspiring increased interest and research efforts on the issues arising from superintelligence, seeding the literature.
Human capital benefits, or network benefits (Will writing this paper help new visiting fellows become familiar with key research domains? Will it help create relationships with outside co-authors? Will it give folks interested in existential risk entry into new communities where valuable contacts may be found?):
- This paper provides an opportunity for networking and collaboration with machine ethics researchers.
Donate Online
Credit card transactions are securely processed through PayPal.