The Singularity Institute Blog

Eliezer Yudkowsky has completed a highly anticipated paper on Timeless Decision Theory, formalizing a proposal for a new decision theory built to avoid the failures of the two standard decision theories, evidential decision theory and causal decision theory. It is the position of the SIAI that developing better formal decision theories is crucial to making progress on the problem of Artificial General Intelligence and seed AI. Here is the long abstract of the paper, simply titled “Timeless Decision Theory”:

Disputes between evidential decision theory and causal decision theory have continued for decades, with many theorists stating that neither alternative seems satisfactory. I present an extension of decision theory over causal networks, timeless decision theory (TDT). TDT compactly represents uncertainty about the abstract outputs of correlated computational processes, and represents the decision-maker’s decision as the output of such a process. I argue that TDT has superior intuitive appeal when presented as axioms, and that the corresponding causal decision networks (which I call timeless decision networks) are more true in the sense of better representing physical reality. I review Newcomb’s Problem and Solomon’s Problem, two paradoxes which are widely argued as showing the inadequacy of causal decision theory and evidential decision theory respectively. I walk through both paradoxes to show that TDT achieves the appealing consequence in both cases. I argue that TDT implements correct human intuitions about the paradoxes, and that other decision systems act oddly because they lack representative power. I review the Prisoner’s Dilemma and show that TDT formalizes Hofstadter’s “superrationality”: under certain circumstances, TDT can permit agents to achieve “both C” rather than “both D” in the one-shot, non-iterated Prisoner’s Dilemma. Finally, I show that an evidential or causal decision-maker capable of self-modifying actions, given a choice between remaining an evidential or causal decision-maker and modifying itself to imitate a timeless decision-maker, will choose to imitate a timeless decisionmaker on a large class of problems.

Readers new to decision theories can review the Wikipedia page for an overview of the field.