Research Grants

Academic Paper Grant

The Coherence of Human Goals

Research summary:

Eliezer Yudkowsky's "Coherent Extrapolated Volition" (CEV) proposes a notion of AI safety based on aggregating idealized human preferences. If an AI is to benefit a group of individuals with distinct traits, then in addition to a way to extrapolate such idealized preferences, it also requires a principle of social choice to resolve conflicts. To assess the general CEV strategy, we must evaluate the sensitivity of the algorithm to the preference idealization scheme, to the social choice theory, and to various other factors, like demographic trends and ideological fads. This paper will make a first attempt, briefly reviewing various relevant literatures, and suggesting avenues for further research.

Planned contents include:

  • A discussion of the issue of AI safety, the motivations behind CEV, and the problems of incomplete and conflicting human preferences (stated values, practical behavior, and otherwise).
  • Brief discussion of some relevant research:
    • historical social/economic pressures on human morality.
    • human universals and differences in moral psychology.
      •
    • reflective equilibrium and "ideal observer" theories.
    • the roles of prior beliefs and values in epistemic disagreement.
    • social choice and political theory.
  • An assessment of the merits of CEV in contrast to simpler algorithms as tools to create AI with widely acceptable motives.
  • A discussion of the largest areas of uncertainty, and of where future research might be most valuable.


Prior related work:

Coherent Extrapolated Volition, by Eliezer Yudkowsky.

 

Target dates for:

Extended abstract (Posting an extended abstract on SIAI website, and circulating to a few related academics for comment): 2 weeks after start date.[1]

 

Working paper (Posting a working paper on the SIAI website; circulating to related academics): 6 weeks after start date.

 

Conference submission: 10 weeks after start date.

 

Follow-up steps (Brainstorming, and drafting proposals for, any follow-up publications. Should it be developed into a journal paper?): 12 weeks after start date.

[1] The "starting date" is the date (guaranteed to be within six months of the receipt of grant money) when we have skilled people to allocate to the project.  Extra donations increase our base of skilled people and thereby increase the number of projects we can get to; the lagged start date allows us to find new people, bring them here, and train them.

Total budget:  $4,400
  • Conference fees, air travel, motel: $1,400
  • Costs for researcher time: $3,000

How research costs are estimated:
  • Person-months for research and writing: 1.25 (This is our standard estimate[1] of person-months per conference paper.)
  • Dollars required to support one skilled full time researcher-month[2]: $2,400

[1] Our base estimate is 1.25 person-months per conference paper, and 3 per journal article, for an experienced full-time researcher. This estimate takes the planning fallacy, and the importance of an outside view in avoiding that fallacy, into account. While typical rates of article production by professors are extremely low, the distribution is strongly skewed towards research-oriented universities and departments, and informal surveys of researchers working on existential risks give data consistent with this estimate for full-time work required per paper. Visiting Fellows vary in their experience levels, so that mean productivity is expected to be lower, but a team mix can be selected to account for this.

[2] This billing rate reflects an estimate of financial outlays for SIAI to create the equivalent of one full-time skilled researcher-month, including stipend or hosting expenses, workspace, and administrative or management time, and other supporting expenses. Actual person-months may be greater or lower depending on the labor mix for a particular project, with shortfalls made up from general funds. This rate is not reflective of the money researchers could earn in the competitive labor market. Think of this as a matched donation. You donate the living expenses; our researchers donate the surplus value of their labor.


How this paper will help reduce existential risk:


Research benefits (What ideas will the paper explore?  How will that knowledge reduce existential risk?):
  • If human goals are fundamentally in conflict, and technologies arrive on the horizon that promise the power to fulfill almost any particular human goals, there will be great incentives to engage in conflict over whose goals are fulfilled, and great potential to improve outcomes through means to negotiate and enforce deals.

Influence benefits (What target audience will the paper impact, how?  How will that impact help with existential risk?):
  • If CEV-like measures have highly limited capacity to bypass conflict, it would be helpful to know as soon as possible, so that we can attempt to set up institutions that allow for groups with conflicting values to resolve their conflicts peacefully. On the other hand, if arguments show human goals are strongly convergent under reflection or extrapolation, publicizing these arguments will reduce strife.


Human capital benefits, or network benefits (Will writing this paper help new Visiting Fellows become familiar with key research domains?  Will it help create relationships with outside co-authors?  Will it give folks interested in existential risk entry into new communities where valuable contacts may be found?):

  • Writing this paper will help Visiting Fellows become familiar with Friendly AI issues, moral and political philosophy, and psychology.



Credit card transactions are securely processed through PayPal.

PayPal – Grant Donation

(200 characters max)