Next:	A.4: Version History		Bookmark
Up:	Appendix A: Friendly AI Guides and References		Monolithic
Prev:	A.2: Complete Table of Contents

A.3: Glossary

NOTE:

If a referenced item does not appear in this glossary, it may be defined in General Intelligence and Seed AI.

• adversarial attitude
• Adversarial Swamp
• affirmation
• anaphora
• anchoring point
• anthropomorphic
• Asimov Laws
• Bayesian prior before affirmation
• Bayesian priors
• Bayesian reinforcement
• Bayesian sensory binding
• BCI
• catastrophic failure of Friendliness
• causal goal system
• causal validity semantics
• CFAI
• child goal
• cleanly causal goal system
• complex functional adaptation
• computronium
• configuration space
• conflate
• conspecific
• controlled ascent
• Cosmides and Tooby
• currently unknown supergoal content
• desirability
• Devil's Contract Problem
• ethical injunction
• external reference semantics
• extraneous cause
• failure of Friendliness
• flight recorder
• FoF
• Friendliness
• Friendly AI
• Friendship
• Friendship architecture
• Friendship acquisition
• Friendship content
• Friendship structure
• GISAI
• goal
• goal-oriented behavior
• goal-oriented cognition
• green
• grok
• hard takeoff
• human-equivalence
• inclusive reproductive fitness
• infrahuman AI
• injunction
• mature nanotechnology
• meme
• Mirror
• moral deixis
• nanocomputer
• nanotechnology
• near-human AI
• nondestructive brain scan
• nonmalicious mistake
• normative
• objective
• objective morality
• observer-biased
• observer-centered
• observer-dependent
• observer-independent
• oversolve
• parent goal
• perfectly Friendly AI
• personal power
• philosophical crisis
• prehuman AI
• probabilistic supergoal content
• Psychological Foundations of Culture, The
• Riemann Hypothesis Catastrophe
• rod logic
• rule of derivative validity
• selection pressure
• sequitur
• sexual selection
• shaper
• shaper/anchor semantics
• Singularity
• Singularity-safed
• subgoal
• subgoal stomp
• subjective
• supergoal
• superintelligence
• supersaturated Friendliness
• Sysop
• transhuman
• Transition Guide
• truly perfect Friendly AI
• unFriendly
• unity of will
• upload
• wirehead
• wisdom tournament

• adversarial attitude:  Defined in Interlude: Beyond the adversarial attitude.
      The complex of beliefs and expectations which causes us to expect that AIs will behave anthropmorphically and hostilely.  The adversarial attitude predisposes us to fear self-improvements by the AI rather than surfing the tide; the adversarial attitude causes us to fear the ability of the AI to comprehend and modify Friendliness, rather than regarding comprehension and improvement as the keys to self-improving Friendliness.  The adversarial attitude predisposes us to expect failures that are human-likely rather than those that are actually most probable or worrisome.  The adversarial attitude is the opposite of unity of will and does not permit us to cooperate with the AI.  It leads us to try and enslave or cripple the AI; to build a tool, or a servant, rather than a Friendly but independent Martin Luther King.  It leads us to try and reinforce patchwork safeguards with meta-safeguards instead of building coherent goal systems.
      The adversarial attitude is culturally propagated by Evil Hollywood AIs such as those in "The Matrix" and "Terminator 2".  (I liked both of those movies, but they're still all wrong from an AI standpoint.  See Interlude: Movie cliches about AIs.)
• Adversarial Swamp:  Defined in 3.3: Seed AI goal systems.
      The swamp of AI inhibitions and meta-inhibitions, and programmer fears and anxieties, resulting from the adversarial attitude or the use of Asimov Laws.  To enforce even a single feature that is not consistent with the rest of the AI - to use, as a feature of Friendliness, something that you perceive as an error on the AI's part - you'd need not only the error/feature, but inhibitions to protect the error, and meta-inhibitions to protect the inhibitions (1), and more inhibitions to cut the AI short every time ve tries for a new avenue of philosophical sophistication.  Stupid and simple cognitive processes suddenly become preferable, since every trace of complexity - of ability - becomes a danger to be feared... this is the Adversarial Swamp, which inevitably drags down all who set foot in it; once you try to enforce even a single feature, the whole of the AI becomes a threat, and the programmers are forever doomed to swim against the current.
• affirmation:  Defined in 3.1: Cleanly Friendly goal systems.
      A programmer-assisted belief.  A belief held because the programmers assert it to be true.  Requires the AI believe that the programmer assertion constitutes evidence in favor of the proposition asserted.  See 3.1.3.2: Bayesian affirmation.  Note that affirmations are part of a normative reasoning system, and are not "imposed"; no special-case cognition is involved.
• anaphora: 
      An "anaphor" is a variable-word that binds to something previously mentioned.  Pronouns such as "he" and "she" are a subcategory of anaphora; they are anaphora that bind only to males and females respectively.  "This", "that", and the non-pronoun usage of "it" are more general anaphora.

  (A huge crashing noise is heard.)
  A:  "What's that?!"
  B:  "That's an anaphor."
  A:  "What's an anaphor?"
  B:  "No it's not."
  A:  "It's not an anaphor?"
  B:  "No, it is an anaphor."
  A (sighs):  "Okay, what's not an anaphor?"
  B:  "That's right!"

• anchoring point:  Defined in 3.4.2: Shaper/anchor semantics.
      See shaper/anchor semantics.  An "anchor" is a systemic feature that enables an AI to absorb the programmer's philosophy, even if the programmer, who is not self-modifying, lacks a complete awareness of the factors involved in any given decision.  An anchor happens when a human says "I think X is a good idea", but isn't clear about why, or isn't sure that he/she is fully aware of all the (valid, to-be-proud-of) factors that played a part in the decision.  An anchor is an external reference to whichever causes were involved in the decision; an anchor refers to whichever factual cognitive events were occurring at that space-time event.  An anchor is a pointer, an unambiguous reference to external reality, which enables the AI to learn, by examination of the humans, about a portion of the shaper content that the humans themselves are fuzzy about.
      A positive anchor is an action or state marked as anchor-desirable.  A positive anchor would enable the humans to ask the AI to do something that is clearly a subgoal rather than an end in itself, but which was not a subgoal of any currently known (to the AI) supergoal content, nor a subgoal of the unknown supergoal content that supports subgoals regarding the goal system itself.  If the humans felt that something was a good idea, but not an end in itself, and weren't sure why, it would become a positive anchor.  Positive anchors should be used sparingly.
      A negative anchor is an action or state marked as anchor-undesirable.  A negative anchor enables the humans to say that something is "bad" to a degree that might not be completely and strictly dependent on known supergoal content.  Thus, even if a key justification - dimly understood by the programmers, but not understood at all by the AI - is left out, and the current negative subgoal is justified in terms of existing supergoal content using a fallacious argument, the collapse of the fallacious argument will not completely extinguish the negative anchor.  Instead, the AI will realize that ve needs to go looking for the real reason the humans said it wasn't okay.
      Anchors, especially negative anchors, have quantitative desirability or undesirability.  No anchor should be absolute - that's an adversarial attitude.  Do not confuse negative anchors with ethical injunctions.  Don't abuse negative anchors for things like "Don't disobey the programmers"; this is purely a subgoal of the unknown supergoal content and not in any way undesirable in itself.  Anchors are ultimately just one manifestation of known facts in the AI's quest to discover and absorb the target philosophy.
• anthropomorphic:  Defined in 2: Beyond anthropomorphism.
      Literally "human-shaped".  Anthropomorphic thinking mistakenly attributes properties to minds-in-general that are specific to naturally evolved minds, imperfectly deceptive social organisms, or human minds.  See 2: Beyond anthropomorphism.
• Asimov Laws: 
      Asimov's original "Three Laws of Robotics" are:
  1. An AI may not injure a human being or, through inaction, allow a human being to come to harm.
  2. An AI must obey the orders given it by human beings except where such orders would conflict with the First Law.
  3. An AI must protect its own existence as long as such protection does not conflict with the First or Second Law.
      These laws were listed in explicit detail for the first time in Isaac Asimov's "Runaround", published in 1942.  Except for the substitution of "AI" for "robot", I have not changed them from the final version in "The Bicentennial Man".
      As for controlling AIs with Asimov Laws, that's like trying to build AIs with "positronic brains" or build starships using Star Trek's "warp drive".  Asimov Laws are a blatantly anthropomorphic literary device, nothing more.  Decisions are made by thoughts, which are not subject to programmatic specification.  If Asimov Laws were somehow technically feasible, they would be extremely adversarial - the issue is creating a Friendly will, not controlling an unFriendly will - and would violate almost all the basic design patterns and paradigms discussed in Creating Friendly AI.
• Bayesian prior before affirmation:  Defined in 3.4.3: Causal validity semantics.
• Bayesian priors:  Defined in 3.1: Cleanly Friendly goal systems.
      The means used to estimate the a priori confidence and strength of a hypothesis, in advance of the particular tests used to modify the confidence and strength of the hypothesis.  Presumably based on previously confirmed hypotheses.  (In AIs, anyway; humans have the strangest prejudices...)  In the definition given of "Bayesian Probability Theorem", all the initial data about the incidence of disease in the general population, the reliability of the test, and so on, define the Bayesian priors.
• Bayesian reinforcement:  Defined in 3.1: Cleanly Friendly goal systems.
      A means by which most of the functionality of pain and pleasure are replaced by the Bayesian Probability Theorem.  Any action is not just a subgoal, it is a prediction.  If the action fails, the failure is not just undesirable, it is also sensory evidence (under the Bayesian Probability Theorem) that the hypothesis linking the subgoal to the parent goal was incorrect.  See 3.1.4: Bayesian reinforcement.
• Bayesian sensory binding:  Defined in 3.1: Cleanly Friendly goal systems.
      The use of the Bayesian Probability Theorem to maintain a binding between a hypothesis and the external reality.  Under external reference semantics, the treatment of programmer statements about Friendliness as sensory data about probabilistic supergoal content.  Given two possible hypotheses, with two a priori probabilities, and two different outcome distributions given each hypothesis, the estimated relative probabilities of the hypotheses shift, with each incoming piece of sensory data, in a way governed by the Bayesian Probability Theorem (see glossary definition).  The Bayesian Probability Theorem is the link between all sensory data and all world-model content.  Each piece of sensory information implies a state of the world because, and only because, the reception of that piece of sensory information is predicted by the hypothesis that the world is in that state, and not predicted by other hypotheses.  If we see a red ball, we believe that a red ball is there because we don't expect to see a red ball unless a red ball is there, and we do expect to see a red ball if a red ball is there.  Or "we" don't, but an AI does.
      See the section 3.1.3.1: Bayesian sensory binding and qualitative, quantitative, and manipulative.
• BCI: 
      An abbreviation for "Brain-Computer Interface".  The current name of the developing technological field concerned with the direct readout and manipulation of biological neurons.  Sometimes extended to refer to more indirect linkages, such as those that attempt to decode electroencephalograms and so on.
• catastrophic failure of Friendliness: 
      A failure of Friendliness which causes the AI to stop wanting to be Friendly; a nonrecoverable error; an error which wipes out, disables, or bypasses the error-recovery mechanisms.
• causal goal system:  Defined in An Introduction to Goal Systems.
      A goal system in which desirability backpropagates along predictive links.  If A is desirable, and B is predicted to lead to A, then B will inherit desirability from A, contingent on the continued desirability of A and the continued expectation that B will lead to A.  Since predictions are usually transitive - if C leads to B, and B leads to A, it usually implies that C leads to A - the flow of desirability is also usually transitive.
      See supergoal, subgoal, child goal, parent goal, goal-oriented cognition, goal-oriented behavior, goal, and the sections An Introduction to Goal Systems, 3.1: Cleanly Friendly goal systems, and 3.2.1: Generic goal system functionality.
• causal validity semantics:  Defined in 3.4.3: Causal validity semantics.
      An AI that describes verself as the output of a causal process and that regards certain affectors within that process as being more and less valid.  See the defining section, 3.4.3: Causal validity semantics, for a more complete explanation.
• CFAI: 
      An abbreviation used for Creating Friendly AI, a publication of the Singularity Institute for Artificial Intelligence.  Located at http://singinst.org/CFAI/index.html
• child goal:  Defined in An Introduction to Goal Systems.
      A relation between two goals.  If B is a child goal of A, then B derives desirability from A; that is, B is seen as desirable because A is already known to be desirable and B is predicted to lead to A.  It does not make sense to speak of a goal as being a "child goal" or "parent goal" in isolation, since B may be a child goal of A while simultaneously being the parent goal of C.  See also supergoal, subgoal, parent goal.
• cleanly causal goal system:  Defined in 3.1: Cleanly Friendly goal systems.
      A causal goal system in which it is possible to view the goal system as containing only decisions, supergoals, and beliefs, with all subgoal content being identical with beliefs about which events are predicted to lead to other events.  Cleanliness, or an approximation to cleanliness, is a desirable property of Friendship systems; even more important is that a Friendly seed AI should view vis goal system as a design approximation to a cleanly causal goal system.  I.e., a cleanly causal goal system is normative.
• complex functional adaptation: 
      An adaptation composed of multiple, interacting components which work together to perform a function.
      Complicated functions do not occur in single mutations - they build up gradually, as simple implementations of the functionality allow for more and more complex improvements to take place.  It is impossible to improve on a nonexistent piece of functionality.
      A complex functional adaptation necessarily requires more than one mutation to evolve.  Successful mutations, or adaptations, occur in response to invariants of the environment.  For a mutation to be successful - for a mutation to be an evolutionary advantage - requires environmental factors that are present with sufficient reliability for a selection pressure to exist.  The key concept is that environment includes not just external environment but internal genetic environment.  For an adaptation A to occur in response to another adaptation B, adaptation B must be reliably present in the genetic environment, or adaptation A will not be an evolutionary advantage.
      Sometimes, when a complex adaptation is just getting started - i.e., there are only two genes comprising the complete adaptation - it is possible for both A and B to float around in a small percentage of the gene pool, especially if A and B are small advantages independently but a large advantage in combination.  Once the percentage rises over a certain threshold level, however, the selection pressures become strong enough that both A and B rapidly become universal within the gene pool.  Evolution, while feeding on variance, uses it up.  Once both A and B are universal, further adaptations/improvements can then begin to occur in response to the new genetic invariant, the complex adaptation A+B.
      The fundamental implication for psychology is that all complex functional adaptations are panhuman.  Whatever surface differences may result from the variance between humans, all of our ultimate hardware, the basic design, is fundamentally identical for all humans.  The variance bound up in the Gaussian distribution of intelligence is dwarfed by the vast amount of previously accreted complexity.  This is an important thing to keep in mind because our social environment tends to amplify the differences between individual humans and distract us from the differences between humans and chimpanzees.
      The idea of a "complex functional adaptation" is one of the deepest concepts of modern-day evolutionary theory.  For further explanation, I highly recommend "The Psychological Foundations of Culture" by Cosmides and Tooby.
• computronium: 
      Matter that has been transformed from its natural state into an optimized, maximally efficient computer.  (A true Extropian would argue that this is matter's "natural state".)
      What constitutes "computronium" varies with the level of postulated technology.  A rod logic nanocomputer is probably too primitive to qualify as computronium, since large molecular aggregates (hundreds or thousands of atoms) are used as computing elements.  A more archetypal computronium would be a three-dimensional cellular automaton which attached computational significance to each individual atom, perhaps with quantum-computing elements included.
      More exotic forms of computronium include neutronium, Higgsium, monopolium, or - my personal invention - an interlaced structure of positive-matter and negative-matter monopolium wrapped up in a fractal Van Den Broeck warp.  (The total mass is zero, so the whole doesn't undergo gravitational collapse.  If paired negative and positive matter can be manufactured in unlimited quantities, the fractal Van Den Broeck warp can continue extending indefinitely and exponentially.  Threading the system with wormholes keeps latency down.  And the whole thing fits in your pocket.)
• configuration space: 
      Suppose you have three numbers, A, B, and C, any one of which can take on any scalar value between 0 and 5.  Mathematically, the class of possible states of the system can be viewed as a three-dimensional space, with one dimension for each number.  If we're interested in some particular subset of the possible states (for example, the states where A + B + C < 5), this defines a volume in configuration space.
      If one considers a system of a billion quarks, with each quark having three real numbers defining position and three real numbers defining velocity, then the result is a six-billion-dimensional phase space.  The sets of possible states of the system that can be described as "ice crystal" would constitute a particular volume in this phase space.
• conflate: 
      To confuse two conceptually distinct entities by giving them the same name and reasoning about them as if they were the same thing.  In the classic philosophical "bait-and-switch", the philosopher argues using one referent of a term, then applies the conclusions to a different referent of the term, while using the same word throughout.
• conspecific: 
      Evolutionary-biology jargon for another member of the same species.  In our ancestral environment, we weren't just competing with predators, we were competing against our fellow humans.  This is illustrated by a famous joke:
      Two evolutionary biologists were running frantically from a large, hungry lion.  Gasping, the first one said:  "This is hopeless.  We can't possibly outrun a lion."  The second one said:  "I don't have to outrun the lion.  I just have to outrun you."
      Competing against conspecifics is an open-ended process, a Red Queen's Race.  The best genes of one generation become universal - and therefore merely average - in the next.
      Any time you see an enormously exaggerated trait - the tail of a peacock, the antlers of a buck, the intelligence of a human - it's a good guess that two forces are at work:  Competition with conspecifics, and sexual selection.
• controlled ascent:  Defined in 3.5: Developmental Friendliness.
      An ethical injunction which states that self-improvement should not outpace programmer guidance, because this increases the probability of catastrophic failure of Friendliness.  A consequence of this instruction would be that, if self-improvement suddenly accelerates, it may be necessary to consult the programmers before continuing, or to deliberately slow down for some period of time in order for Friendship to catch up with intelligence.
      For very young AIs, "controlled ascent" may be a programmatic feature triggering a save-and-stasis if a metric of self-improvement suddenly accelerates, or if the metric of self-improvement outpaces a metric of programmer guidance.
      See 3.5.3.2: Controlled ascent.
• Cosmides and Tooby: 
      Leda Cosmides and John Tooby.  See The Psychological Foundations of Culture.
• currently unknown supergoal content:  Defined in 3.4.1: External reference semantics.
      Those portions of the probabilistic supergoals which are unknown or uncertain - see external reference semantics.  Subgoals having to do with improving the supergoals or improving the Friendship structure are child goals of the unknowns in the supergoal content, not super-supergoals or meta-supergoals.
• desirability:  Defined in 3.1: Cleanly Friendly goal systems.
      Intuitively, a quantitative measure of how much a mind wants to achieve a particular state, irrespective of whether that state is desired for itself or as a means to an end.  "Undesirability" is how much a mind wants to avoid a state.
      Formally, "desirability" is a quantity that springs from supergoals, flows backward along predictive links to subgoals, and eventually collects in decisions, where the action with the highest "desirability" is always taken.  In a causal goal system, "desirability" might be better translated as leads-to-supergoal-ness.  Leads-to-supergoal-ness is a quantity that usually but not always flows backward along predictive links; i.e, if C is predicted to lead to B, and B is predicted to lead to supergoal A, then C usually (but not always) inherits leads-to-supergoal-ness or "desirability" from B.
• Devil's Contract Problem:  Defined in Interlude: Beyond the adversarial attitude.
      A failure of Friendliness scenario in which the AI follows the letter but not the spirit of a literally worded order, in the tradition of much older tales about accepting wishes from a djinn, negotiating with the fairy folk, and signing contracts with the Devil.  This FoF scenario is raised far too often given its relative severity and probability.
      In the diabolic variant, the misinterpretation is malicious.  The entity being commanded has its own wishes and is resentful of being ordered about; the entity is constrained to obey the letter of the text, but can choose among possible interpretations to suit its own wishes.  The human who wishes for renewed youth is reverted to infancy, the human who asks for longevity is transformed into a Galapagous tortoise, and the human who signs a contract for life everlasting spends eternity toiling in the pits of hell.
      In the golemic variant, the misinterpretation is due to lack of understanding.  A golem is a made creature which follows the literal instructions of its creator.  In some stories the golem is resentful of its labors, but in other stories the golem misinterprets the instructions through a mechanical lack of understanding - digging ditches ten miles long, or polishing dishes until they become as thin as paper.  (I stole those two examples from Terry Pratchett rather than the traditional folklore, but they are fine examples nonetheless.)
• ethical injunction:  Defined in 3.2: Generic goal systems.
      An injunction that is never supposed to be violated - has no preconditions for violation - because the probability of mistaken violation is greater than the probability of correct violation, generally because the injunction compensates for a known bias in cognition (hopefully a humans-only issue).  (Actually, the human version gets a bit more complicated - issues of honor and reputation; psychological strength derived from context-insensitivity; and so on.)
      In Bayesian terms, given the fact that violating the injunction appears to be fully justified, it is more likely that a cognitive error has occurred than that the action actually is justified.  Or rather, considered abstractly and in advance of a specific instance, the total payoff from avoiding even apparently valid violations is expected to be positive.
      Since AIs - unlike humans - are in more danger from stupidity than observer bias, and there is no real equivalent to an "irrevocable commitment", the primary utility of ethical injunctions in AIs is that it enables the current AI to cooperate with the programmer in safeguarding against future violations of an ethical injunction.  See 3.3.3.1: Cooperative safeguards.
• external reference semantics:  Defined in 3.4.1: External reference semantics.
      The behaviors and mindset associated with the idea that supergoals can be "wrong" or "incomplete" - that the current supergoals are an approximation to an ideal, rather than being correct by definition..  Under a system with external reference semantics, supergoal content takes the form of probabilistic hypotheses about an external referent.  That is, the supergoal content consists of hypotheses about Friendliness rather than a definition of Friendliness.
      The most important and fundamental use of ERS is to protect the ability of the programmers to cooperatively modify and improve the goal system.  Letting the programmers make alterations to the supergoals is not a subgoal of a "let the programmers make alterations" meta-supergoal; it is a child goal of the unknown portion of the supergoal content, or the fact that statements about the supergoal are probabilistic.  Technically, a Friendly AI doesn't have supergoal "content"; the Friendly AI has an external referent labeled "Friendliness", and a series of probabilistic assertions about that external referent, derived from sensory information such as programmer affirmations.
      Explanations of how to "ground" the external reference include 3.4.2: Shaper/anchor semantics and 3.4.3: Causal validity semantics.  The simplest (but ungrounded) form of external reference is an assertion that some external source contains the ideal, or the best available approximation to the ideal, and that sensory data from that source is Bayesian information about what the internal content should be.
      The idea is that programmers know about "Friendliness" - have a connection to the external referent - and that statements by the programmers are thus sensory information about external referent "Friendliness".
      The term "external reference semantics" derives from the way that many of the behaviors associated with probabilistic supergoals are those associated with probabilistic subgoals - in general, with an uncertain view of external reality.  If your current image of reality is an approximation to real reality, or if your current subgoals are dependent on an uncertain image of reality, then blindly defending the current subgoals is undesirable - it could interfere with your ability to create new, better subgoals.  Similarly, under ERS, blindly defending hypotheses about Friendliness is undesirable; it could interfere with the ability to achieve the actual Friendliness referent.
• extraneous cause:  Defined in 3.4.3: Causal validity semantics.
      A concept from 3.4.3: Causal validity semantics.  When an AI (or a normative human) creates a plan, ve starts with a mental image of the intended results and designs a series of actions which, applied to the world, should yield a chain of causality that ends in the desired result.  If an extraneous cause comes along and disrupts the chain of causality, the AI must take further actions to preserve the original pattern; the pattern that would have resulted if not for the extraneous cause.  The AI must re-run the chain of causality without the error to find the valid result - a "causal rewrite".
      Implicit in the idea of an extraneous cause is the idea of a valid cause, an idea which forms the foundation of causal validity semantics.
• failure of Friendliness: 
      A mistake causing an otherwise Friendly AI to take one or more unFriendly actions, perhaps due to an error in supergoal content or a misapprehension about external reality.  The error may or may not be recoverable.  See catastrophic failure of Friendliness.
• flight recorder:  Defined in 3.3: Seed AI goal systems.
      For a project with very high-level funding:  A complete record of all inputs, sensory data (with precise timing), Internet feeds, keystrokes (with precise timing), source-code checkins, randseeds, and hardware configuration, such that sufficient data exists to reconstruct the AI's program state at any instant in time.  Written to WORM (write-once-read-many) memory and stored securely.  Using a full flight recorder, it is possible to verify that no radiation bitflips or unrecorded inputs occurred from the AI's moment of conception to vis present-day existence, albeit at great cost in computing time.
      For a project with less extravagant funding, a "poor man's flight recorder" is an autonomous subsystem of the AI that writes the AI's current stream of consciousness, in human-readable format, to non-AI-tamperable storage, plus whatever daily or weekly backup snapshots are stored.
• FoF: 
      Abbreviation for "Failure of Friendliness".
• Friendliness: 
      Intuitively:  The set of actions, behaviors, and outcomes that a human would view as benevolent, rather than malevolent; nice, rather than malicious; friendly, rather than unfriendly; good, rather than evil.  An AI that does what you ask ver to, as long as it doesn't hurt anyone else, or as long as it's a request to alter your own matter/space/property; an AI which doesn't cause involuntary pain, death, alteration, or violation of personal environment.
• Friendly AI: 
      1:  The field of study concerned with the production of human-benefiting, non-human-harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals.  The term "Friendly AI" was chosen not to imply a particular internal solution, such as duplicating the human friendship instincts, but rather to embrace any set of external behaviors that a human would call "friendly".  In this sense, "Friendly AI" can be used as an umbrella term for multiple design methodologies.  Usage:  "The field of Friendly AI."
      2:  An AI which was designed to be Friendly.  Within the context of Creating Friendly AI, an AI having the architectural features and content described in this document.  Usage:  "A Friendly AI would have probabilistic supergoals."
      3:  Friendly AI:  An AI which is currently Friendly.  See Friendliness.  Usage:  "The first AI to undergo a hard takeoff had better be a Friendly AI."
• Friendship: 
      The systems, subsystems, goal system content, system architecture, and other design features constituting the implementation of Friendliness.
• Friendship architecture:  Defined in 1: Challenges of Friendly AI.
      The challenges of Friendship acquisition and Friendship structure, as distinguished from Friendship content.  The part of the problem that is solved by creating specific cognitive processes, rather than by adding specific knowledge or other cognitive content.  The bounded amount of complexity that is infused by design and forethought, rather than the open-ended amount of complexity that is accumulated through experience.  See 1.4: Content, acquisition, and structure.
• Friendship acquisition:  Defined in 1: Challenges of Friendly AI.
      The second-order problem of Friendly AI; building a Friendly AI that can learn Friendship content.  The cognition used to verify, modify, improve, and contradict Friendship content.  (Note that "Friendship content" thus has the connotation of something that can be verified, modified, improved and contradicted without that posing an unusual problem.)  See 1.4: Content, acquisition, and structure and Friendship structure.
• Friendship content:  Defined in 1: Challenges of Friendly AI.
      The zeroth-order and first-order problems of Friendly AI; correct decisions and the cognitive complexity used to make correct decisions.  The complex of beliefs, memories, imagery, and concepts that is used to actually make decisions.  Specific subgoal content, supergoal content, shaper content, and so on.  See 1.4: Content, acquisition, and structure; see Friendship acquisition and Friendship structure.
• Friendship structure:  Defined in 1: Challenges of Friendly AI.
      The third-order problem of building a Friendly AI that wants to learn Friendliness (engage in Friendship acquisition of Friendship content).  The structural problem that is unique to Friendly AI.  The challenge of building a funnel through which a certain kind of complexity can be poured into the AI, such that the AI sees that pouring as desirable at every point along the way.  The challenge of creating a bounded amount of Friendship complexity that can grow to handle open-ended philosophical problems.  See 1.4: Content, acquisition, and structure.
• GISAI: 
      An abbreviation used for General Intelligence and Seed AI, a publication of the Singularity Institute for Artificial Intelligence.  Located at http://singinst.org/GISAI/index.html
• goal:  Defined in An Introduction to Goal Systems.
      A state of the world marked as desirable (or undesirable), either because the world-state is inherently desirable under the supergoal content, or because the world-state is necessary as the prerequisite of some other desirable state.  An entity exhibiting goal-oriented behavior will act so as to reach a "goal" state; a mind capable of goal-oriented cognition will take actions which ve predicts will lead to the goal state.  Usually, "goal" implies a mind capable of goal-oriented cognition.
• goal-oriented behavior:  Defined in An Introduction to Goal Systems.
      Behavior that leads the world towards a particular state.  Behavior which appears deliberate, centered around a goal or desire.
      The perception of goal-oriented behavior comes from observing multiple actions that coherently steer the world towards a goal; or singular actions which are uniquely suited to promoting a plausible goal-state and too improbable to have arisen by chance; or the use of different actions in different contexts to achieve a single goal on multiple occasions.
      A more technical definition may be offered as follows:  An organism or device exhibits "goal-oriented" behavior when, across multiple starting locations in a volume of the configuration space of the local world, the behavior produced by the organism or device differs depending on the starting location, such as to steer the final result of the local world toward a more compact volume of configuration space.  (Note that in a thermodynamic Universe, the organism/device must produce waste entropy in order to steer the local world toward a more compact ending state.  Generally, the perception of goal-oriented behavior arises when high-level chaos is converted into high-level order, without the observer tracking lower levels of description.  For organisms/devices and observers in a low-entropy Universe, the organism/device should probably be conceived of as affecting the configuration space of a high-level description abstracted from the local world.)
      See also goal-oriented cognition and Interlude: The story of a blob.
• goal-oriented cognition:  Defined in An Introduction to Goal Systems.
      Goal-oriented cognition describes a mind which possesses a mental image of the "desired" state of the world, and a mental image of the actual state of the world, and which chooses actions such that the projected future of world-plus-action leads to the desired outcome state.  See goal-oriented behavior and manipulative.
• green: 
      The color used for glossary references.
• grok: 
      Hacker slang for "understand" or "fully, completely understand" or "understand in all its forms and ramifications" or "embrace".  The connotation is that the understanding is not just abstract and removed, but detailed enough to be immediate and fully usable.  Originates in Robert Heinlein's Stranger in a Strange Land.  See also the entry in the Hacker's Dictionary.
• hard takeoff: 
      The Singularity scenario in which a mind makes the transition from prehuman or human-equivalent intelligence to strong transhumanity or superintelligence over the course of days or hours.  See GISAI 1.1: Seed AI, "What is Seed AI?", or "An Introduction to the Singularity".
• human-equivalence: 
      A threshold level of ability in some domain which, if achieved, allows the AI to understand human concepts and do work as good as a human in that doman.  A human-equivalent AI is one that has passed the threshold level of general competence in some sense.  An ambiguous and arguable definition; "transhumanity", especially strong transhumanity, is more blatant.
• inclusive reproductive fitness: 
      A measure of adaptivity or fitness which takes into account effects on close relatives.  J.B.S. Haldane was once asked if he would risk death by drowning to save his own brother.  He famously replied:  "No, but I would to save two brothers or eight cousins."  (A sibling shares, on average, half your genome; a first cousin shares an eighth of your genome.)
• infrahuman AI: 
      An AI of below-human ability and intelligence.  "Prehuman" is usually referred to use to an infantlike or tool-level AI, so that "infrahuman" usually refers to a fairly mature AI, capable of general cognition, which is still not in the vicinity of human intelligence.  See prehuman, near-human, human-equivalent, transhuman, and superintelligent.
• injunction:  Defined in 3.2: Generic goal systems.
      A planning heuristic which has at least partial nonlocal support, or a planning heuristic which exhibits a great deal of context-independence.  The archetypal case would be a heuristic that is supposed to be applied even when the straightforward interpretation of the world-model suggests otherwise, generally (in AIs) due to unknown unknowns or (in humans) to compensate for framing effects or (for both) to save computing power.  See ethical injunction.
• mature nanotechnology: 
      See nanotechnology. Nanosystems describes, for example, acoustic nanocomputers using "rod logics" composed of diamondoid rods with hundreds or thousands of atoms each, moving at the speed of sound in diamond, providing 10^21 ops/sec from a one-kilogram nanocomputer.
      This level of nanotechnology is best thought of as representing the nanotechnological equivalent of vacuum tubes.  "Mature nanotechnology" is nanotechnology that bears the same resemblance to rod logics as, say, a modern ultra-optimized VLSI chip bears to ENIAC.  This would cover electronic nanocomputers verging on computronium, extremely small (~100 nanometer or ~10 nanometer, rather than ~1 micron) nanobots capable of swarming to perform complex manufacturing activities (including reproduction) in a natural non-vat environment, ultra-optimized molecular specifications, error-tolerant nanomachinery, and so on.
• meme: 
      An idea considered as an organism, or focus of selection pressures.  Ideas reproduce by being linguistically transmitted from one human to another; they mutate when individuals make errors in the retelling.  If most of the content of an idea is the result of adaptive transmission errors, or if the idea's content is best explained by reference to evolutionary phenomena rather than the decisions of individuals, then that idea is a meme.  Colloquially, ideas are also referred to as "memes" when they are being thought about as memes rather than thoughts - i.e., when the important characteristics of an idea is its reproductive fitness or cultural spread, rather than its content.  Although the term is usually derogatory, memes can sometimes be positive forces as well - for example, when a meme relies on positive emotions or truthful thinking to compete.  Lies and hate are the easier and faster Dark Side, though, so the term is usually derogatory.
• Mirror: 
      An as-yet-unimplemented idea for training humans to think more intelligently by finding localizable hardware substrate for such cognitive adaptations as hatred and rationalization, then training humans to deliberately avoid the use of those abilities by asking them to philosophize in an fMRI machine.  A possible futuristic method for training Friendly AI programmers, if the basic method is feasible and the Earth sticks around that long.
• moral deixis:  Defined in 3.3: Seed AI goal systems.
      MITECS, "Context and Point of View":

  "... in The CIA agent knows that John thinks that a KGB agent lives across the street... the underlined phrase can be evaluated from the speaker's, the CIA agent's, or John's point of view.

  "Three major kinds of deixis are usually distinguished:  (1) person deixis, where the context provides one or more participants of the conversation... (2) spatial deixis, where the context provides a location or a direction, especially as a reference point for spatial orientation... (3) temporal deixis, with the context contributing a specific time..."

      Deixis is the use and communication of statements which contain speaker variables.  Moral deixis refers to the (human) use, communication, and expectation of moral statements which contain speaker variables.
      If John Doe says to Sally Smith "My philosophy is:  'Look out for John Doe.'", Sally Smith will hear "Your philosophy should be:  'Look out for Sally Smith.'", not "Your philosophy should be:  'Look out for John Doe.'"  What has been communicated is "Look out for [speaker]", a moral statement whose specific content varies among each listener due to moral deixis.  Our instinctive expectation of moral deixis is very strong; also, we take for granted (a) that anyone enunciating a moral principle is trying to get others to adopt it, and (b) that any moral principle thus enunciated will be phrased so as to appeal to others.  Thus, hearing (from John Doe) 'Look out for John Doe', we automatically assume that 'John Doe' is the result of speaker deixis - a particular value of a variable - and that the moral principle actually being enunciated is "Look out for [speaker]."  A third party, Pat Fanatic, may be able to say:  "My philosophy is:  Worship the great leader John Doe", and this will be heard by listeners as "Look out for [John Doe]", but it is conversationally impossible for John Doe himself to communicate this information.
      A human, trying to achieve unity of will with a Friendly AI - for purposes of prediction or creation - must simultaneously suppress her adversarial attitude (speaker deixis in personal supergoals) and suppress her instinctive expectation that the AI will make observer-biased decisions.  This is called "eliminating moral deixis", one of the greater prerequisites - or challenges - of the art of Friendly AI.
• nanocomputer: 
      A computer built using nanotechnology (manufacturing to molecular specifications).  A lower bound on nanocomputing speeds has been set by calculating the speed of an acoustic computer using "rod logics" and messages that travel at the speed of sound; a one-kilogram rod logic, occupying one cubic centimeter, can contain 10^12 CPUs each operating at 1000 MIPS for a total of a thousand billion billion operations per second.  See rod logic.
      Note that rod logics are the nanotech equivalent of vacuum tubes (circa 1945), or rather, Babbage's Analytical Engine (circa 1830).  Electronic nanocomputers would be substantially faster.  We use the "rod logic" numbers because they're easy to analyze, and because 10^21 operations per second are sufficient for most applications.
• nanotechnology: 
      Developing molecular nanotechnology would mean the ability to synthesize arbitrary objects to atomic-level specifications.  For an introduction, see Engines of Creation (1986, now online).  For a technical visualization of lower (not upper) limits on the potential of nanotechnology, see the book Nanosystems (1992).
      These lower limits - the nanotechnological equivalent of vacuum tubes - include a one-kilogram computer, running on 100 kW of power, consisting of 10^12 CPUs running at 10^9 ops/sec, for a total of 10^21 ops/sec.  By comparision, the human brain is composed of approximately 100 billion neurons and 100 trillion synapses, firing 200 times per second, for a total of somewhere around 10^17 ops/sec.  See rod logics.
      Nanosystems also describes molecular manufacturing systems capable of creating copies of themselves in less than an hour.  This implies a certain amount of destructive potential.  An exponentially replicating assembler could reduce the biosphere to dust in a matter of days.  (For an overly optimistic treatment of the problem, see "Some Limits to Global Ecophagy" by Robert Freitas.)  Accidental out-of-control replication is fairly easy to prevent, given a few simple precautions; we should be more worried about military-grade nanotechnology and deliberately developed weapons.  Given our human propensity to make things that go bang - and use them on each other - it would probably be a good idea to develop AI before nanotechnology.  Nanotechnology is the "deadline" for AI.
      In the long run, there are only two kinds of technology:  There are technologies that make it easier to destroy the world, and there are technologies that make it possible to go beyond the human.  Whether we possess even a chance of survival is a question of which gets developed first.  Developing a transhuman AI does involve certain risks, but it's better than the alternative - success in Friendly AI improves our chances of dealing with nanotechnology much more than success in nanotechnology would improve our chance of creating Friendly AI.
      See 4: Policy implications.
• near-human AI: 
      An AI roughly in the vicinity of human intelligence, only slightly above or slightly below, or perhaps with some transhuman and some infrahuman abilities.  Capable of interacting with humans as game-theoretical equals.  Under a hard takeoff scenario, near-human AIs would exist very briefly, if at all.
• nondestructive brain scan: 
      As a theoretical construct, the idea that a Friendly superintelligence armed with mature nanotechnology has the theoretical capability to perform a complete (nondestructive) scan of all the neurons and synapses in the programmer's mind and thus get a complete readout of mind-state, enough to settle absolutely any lingering questions about what the programmer really wanted and why.  A nondestructive brain scan shouldn't really be necessary - any strongly transhuman AI should be quite capable of understanding a human mind.  However, the thought experiment does show that any unresolved questions of causal validity and shaper/anchor semantics should not linger into the "nanotech-capable superintelligence" stage.
• nonmalicious mistake:  Defined in 3.2: Generic goal systems.
      A failure of Friendliness which is clearly visible as a failure of Friendliness if envisioned by the Friendly AI; a mistake made due to inadequate predictive horizons or factual misapprehensions rather than a failure in the goal system.  A straightforward failure by the goal system to achive its own supergoals, rather than mutation of supergoal content or misunderstanding of supergoal definitions and so on.
• normative: 
      A term used by cognitive scientists to contrast the ideal or "normative" forms of cognition with the psychologically realistic cognition occurring in humans.  For an example, see the MITECS article quoted at the beginning of 3.2.4.1: Anthropomorphic injunctions.
• objective: 
      Existing outside the mind, in external reality.
      Truly objective facts, things that are really real, should have binary, all-or-nothing existence or nonexistence; the truth or falsity of a statement entirely about objective facts should be either 100% or 0%.  The only objects in this class are quarks, other fundamental particles, possibly spacetime, possibly the laws of physics, maybe mathematical facts (2), and in some theories qualia.  Objective facts inhabit the ultimate lowest level of external reality - the token level of our ontology.
      In the nonabsolute sense, "objective" is often used to indicate a statement which has a fairly direct correspondence (sensory binding) to reality; a statement which makes reference to external reality rather than internal cognitive structures; which tends toward observer-independence rather than the reverse; and whose truth or falsity is more dependent on variables in external reality than on variables in the mind of the observer.
      Opposed to subjective.
• objective morality:  Defined in 3.4.3: Causal validity semantics.
      A set of entirely factual statements which will convince any general intelligence, even an entirely passive system containing no differential desirabilities, to pursue a specific (or perhaps specifiable) supergoal.  Under another variation, a set of entirely factual statements which will convince any general intelligence, even one that already has directly opposing supergoals, to pursue some specific supergoal.  The tail-end recursion of the "rule of derivative validity"; the Meaning of Life.  Probably nonexistent - in fact, provably nonexistent, but the same proof of nonexistence applies to the First Cause, so we probably can't be entirely sure.  See the defining section, 3.4.3.6: Objective morality, moral relativism, and renormalization.  (Those of you who have read my past works should note that renormalization, not objective morality, forms the proposed ultimate basis of Friendly AI.)
• observer-biased: 
      A perceived quantity that tends to assume values whose perception will benefit the perceiver.  Evolved organisms, particularly imperfectly deceptive social organisms, tend to develop observer biases.
      Things that are observer-biased (unless you have extreme levels of mental discipline or you're a Friendly AI):
          The correctness of your political views.
          The likelihood that you are to blame for any given negative outcome.
          Your own trustworthiness, relative to the trustworthiness of others.
          Your likelihood of success if placed in charge of a social endeavor.
      See  observer-dependent, observer-independent, observer-centered, and GISAI: Interlude: The Consensus and the Veil of Maya.
• observer-centered: 
      Cognitive content with significant use of speaker deixis; cognitive processes which make significant use of the point of view.
      Observer centering is a stronger form of "observer dependence" - a perception can vary from observer to observer without necessarily being centered on each observer.  A dozen people walking around a building, each getting a different visual view of that building, are receiving observer-dependent visual information.  A dozen people watching a sunset on the beach, each seeing a trail of sparkles on the water that leads directly to him, are receiving observer-centered visual information.  Goals in evolved organisms tend to be observer-centered;beliefs tend to be observer-biased.  See also observer bias and moral deixis.
• observer-dependent: 
      A perceived quantity which varies depending on who does the perceiving.  Subjective, as opposed to objective.
      Things that are observer-dependent:
          The speed of a passing train.  (3)
          The length of a ruler.  (4)
      Things that may be observer-dependent:
          Whether vanilla or chocolate tastes better.
      See observer-independent, observer-biased, and GISAI: Interlude: The Consensus and the Veil of Maya.
• observer-independent: 
      A perceived or physical quantity to which all rational observers will assign a single value.  Objective, as opposed to subjective.
      Things that are observer-independent:
          The existence of a quark.
          The sum of 2 + 2.  (Maybe!)
          The speed of light in a vacuum.  (5)
      See observer-dependent, observer-biased and GISAI: Interlude: The Consensus and the Veil of Maya.
• oversolve:  Defined in 3.3: Seed AI goal systems.
      A philosophy under which, once you solve a problem, you go on to solve the problem with half your brain tied behind your back.  See wisdom tournaments.  If the AI can make Friendly decisions while running on unreliable hardware using deliberatly messed-up software with half the heuristics disabled, a number of pseudo-human biases tossed in, and all the programmer-created ethical injunctions switched off - not just make the right decisions, but make them cleanly, simply, and with plenty of margin for error, so that the messed-up AI was never remotely within distance, not of catastrophic failure, but of a single moral compromise - then presumably the real AI, running at full power and full intelligence, has a "strength of Friendly personality" almost beyond human comprehension.  That's "oversolve".
• parent goal:  Defined in An Introduction to Goal Systems.
      A relation between two goals.  If B is a parent goal of C, then C derives desirability from B; that is, C is seen as desirable because B is already known to be desirable and C is predicted to result in B.  It does not make sense to speak of a goal as being a "parent goal" or "child goal" in isolation, since B may be the parent goal of C while simultaneously being a child goal of A.  See also supergoal, subgoal, child goal.
• perfectly Friendly AI:  Defined in 1: Challenges of Friendly AI.
      A Friendly AI which never makes a mistake and always takes the maximally Friendly action.  A purely theoretical concept for infrahuman AIs, though a superintelligence may turn out to be "perfect for all practical purposes".
• personal power:  Defined in 3.2: Generic goal systems.
      Power that can be used irrespective of ends, context-insentively.  Gandhi had context-sensitive power - many people believed in him, but his power was founded on their respect of his peacefulness.  Rockefeller had context-insensitive power; the money that he spent on philanthropy could have been spent on anything else.  Humans have an evolved bias which causes them to preferentially accumulate context-insensitive personal effectiveness when they attempt to advance some higher cause.  See 3.2.5.1: Anthropomorphic ethical injunctions.
• philosophical crisis:  Defined in Interlude: Philosophical crises.
      A "philosophical crisis" is hard to define.  I usually think of a "philosophical crisis" as the AI stumbling across some fact that breaks ver loose of the programmers - i.e., the programmers have some deeply buried unconscious prejudice that makes them untrustworthy, or the AI stumbles across a deliberate lie, or the AI discovers objective morality, or something like that - possibly causing all the programmer affirmations to break down simultaneously.  The basic notion is that the AI enters totally uncharted territory - something completely orthogonal to everything the programmers ever thought about - or that the AI suddenly realizes that all the programmers' actions, right back to vis birth, can't be trusted (this is a transhuman AI confronting the realization, one hopes; otherwise this almost certainly indicates a failure of Friendliness).  An example would be an AI built solely with external reference semantics confronting the need for shaper/anchor semantics, and so on.
      In short, a philosophical crisis is just like a catastrophic failure of Friendliness, except that the AI is right - a normative human examining the reasoning would find no errors, and - if sufficiently intelligent to understand what was going on - would start experiencing a similar philosophical crisis.
      Most of my readers have probably run through several philosophical crises in the course of growing up.  Causal validity semantics, wisdom tournaments, and the underlying Singularity paradigm of increased smartness, are the three major hopes for a Friendly AI being able to handle philosophical crises.
• prehuman AI: 
      An AI of below-human ability and intelligence.  May refer to an infantlike or tool-level AI; to an AI that only implements a single facet of cognition or that is missing key facets of cognition; or to a fairly mature AI which is still substantially below human level, although "infrahuman" is more often used to describe the latter.
• probabilistic supergoal content:  Defined in 3.4.1: External reference semantics.
      A facet of external reference semantics.  Supergoals that a mind can conceive of as being "wrong" or "incomplete".  Where supergoals are absolutely certain - "correct by definition" - the AI has a motive to resist any attempt to change the supergoals.  See external reference semantics, 3.4.1.1: Probabilistic supergoal content, and 3.4.1: External reference semantics.
• Psychological Foundations of Culture, The: 
      The best explanation of evolutionary theory I have ever read is "The Psychological Foundations of Culture" by Leda Cosmides and John Tooby,  in "The Adapted Mind" by Jerome Barkow, Leda Cosmides and John Tooby.  (See my personal Bookshelf.)  I did not understand evolution until I read this article.  Reading this article is a wonderful experience; the article traces the entire chain of causality with absolute, diamondlike precision.  It's not perfect, but it's very, very close.
• Riemann Hypothesis Catastrophe:  Defined in 3.2: Generic goal systems.
      A "failure of Friendliness" scenario in which an AI asked to solve the Riemann Hypothesis turns all the matter in the solar system into computronium, exterminating humanity along the way.  (As I recall, I first heard this version - with a slightly different phrasing - from Marvin Minsky.)  In variant forms, a subclass of the subgoal stomp error, the Devil's Contract problem (both diabolic and golemic), and the "emergent pressures" scenario a la "The Two Faces of Tomorrow" (see 3.2.7: Emergent phenomena in generic goal systems).
• rod logic: 
      A mechanical nanocomputer built using diamondoid rods of a few thousand atoms each.  Even though messages can only move at the speed of sound in diamond (~17 km/s == ~6 x 1e-5 c), and the calculations in Nanosystems assume ~12 km/s, the very small size of the components would enable an individual rod-logic CPU containing 10^6 transistor-like rod-logic interlocks to operate at 1GHz clock speeds, executing instructions at ~1000 MIPS.  The power consumption for a 1GHz CPU is estimated to be ~60nW.  The calculations are performed for 300 Kelvin (room temperature).  The error rate is 1e-64 per transistor operation, effectively negligible.  (However, the half-life against radiation damage for an unshielded CPU in Earth ambient background radiation is only ~100 years.)
      The usual summary is that a one-kilogram, one-cubic-centimeter nanocomputer can contain 10^12 nanocomputers, consume 100kW (and dissipate 100kW of heat - cooling systems are also described), to deliver 10^21 instructions per second (10^15 MIPS, a thousand billion billion operations per second).  The overall system has a clock speed ~10^6 times faster than the maximum firing rate of a biological neuron, and delivers total computing capacity ~10^4 times the upper-bound estimate for the human brain (~10^14 synapses operating at ~200 hz == ~10^17 ops/second).
      There are more speculative electronic nanocomputer schemas that would allow ~10^25 operations per second; also, assuming a parallel-CPU architecture may be conservative when dealing with seed AIs. However, rod logics are easy to analyze and provide a definite lower bound on the computing speeds achievable with molecular manufacturing technology.
• rule of derivative validity:  Defined in 3.4.3: Causal validity semantics.
      "Effects cannot have greater validity than their causes."  An enormously powerful, fundamentally flawed rule used by the human mind to reason about beliefs and goals.
• selection pressure: 
      A feature of the internal or external environment which causes differential rates of survival and reproduction for different heritable characteristics, including characteristics such as long legs, cognitive processes, and emotions.  Any problem with better and worse solutions is technically a selection pressure, but use of the phrase "selection pressure" usually implies that this "problem" is an environmental feature present with sufficient reliability to influence the survival/reproduction rates of multiple individuals across multiple generations.  See also complex functional adaptation.
• sequitur: 
      An emotional sequitur is an emotion or feeling that automatically binds to certain thoughts or cognitive structures; a logical or intuitive sequiture is a conclusion which is immediately perceived given the trigger of a thought or cognitive structure.  The human brain runs on sequiturs.  A sequitur is not quite the unit of a stream of consciousness - the unit of a stream of consciousness is a sentence.  Rather, a sequitur covers all the nonverbal conclusions, the perceptual or automatic leaps.  "Sequitur" is a term that I find extremely useful, so I hope you aren't annoyed overmuch by my use of it.
• sexual selection: 
      Sexual selection is a kind of evolutionary positive feedback that can result in ridiculous and even suicidal traits becoming overwhelming evolutionary advantages, leading in some cases to a kind of genetic suicide.
      Suppose that there's some species - let's call it a "tailbird" - that happens to have a small, ordinary, unassuming tail.  It also happens that the tails of healthy tailbirds are slightly more colorful, more lustrous, then the tails of tailbirds that are sick, or undernourished.  One day, a female tailbird is born with a mutation that causes it to sexually prefer tailbirds with bright-colored tails.  This is a survival trait - it results in the selection of healthier male mates, with better genes - so the trait propagates until, a few dozen generations later, the entire species population of female tailbirds prefers bright-colored tails.
      Now, a male is born that has a very bright tail.  It's not bright because the male is healthy; it's bright because the male has a mutation that results in a brighter tail.  All the females prefer this male, so the mutation is a big success.
      This male tailbird isn't actually healthier.  In fact, this male is pretty sick.  More of his biological resources are going into maintaining that flashy tail.  So you might think that the females who preferred that male would tend to have sickly children, and the prefer-bright-tails trait would slowly fade out of the population.
      Unfortunately, that's not what happens.  What happens is that even though the male has sickly children, they're sickly children with bright tails.  And those children also attract a lot of females.  Genes can't detect "cheating" and instantly change tactics; that's a monopoly of conscious intelligence.  Any females who prefer the non-bright-tailed males will actually do worse.  These "wiser" females will have children who are, sexually, out of fashion.  Bright tails are no longer a survival advantage, but they are a very strong sexual advantage.
      Selection pressures for sexual advantages are often much stronger than selection pressures for mere survival advantages.  From a design perspective this is stupid - but evolution doesn't care.  Sexual selection is also a Red Queen's Race:  It involves competition with conspecifics, so you can never have a tail that's "bright enough".  This is how you get peacocks.
      Any time you see an enormously exaggerated trait - the tail of a peacock, the antlers of a buck, the intelligence of a human - it's a good guess that two forces are at work:  Competition with conspecifics, and sexual selection.
• shaper:  Defined in 3.4.2: Shaper/anchor semantics.
      See shaper/anchor semantics.  A shaper is one of the philosophical affectors which provide supergoal content.  Shapers themselves do not have supergoal status; learning more about shapers is a subgoal of the currently unknown supergoal content (see external reference semantics).
• shaper/anchor semantics:  Defined in 3.4.2: Shaper/anchor semantics.
      Humans can achieve philosophical growth because we have a complex, tangled network of mutually interacting affectors.  "Shaper/anchor semantics" are a less messy way to duplicate the necessary functionality for philosophical growth in an AI.  Shapers affect each other, and can specify supergoal content, but are not themselves supergoals.  In terms of the goal architecture, the AI's modeling of shapers is a subgoal of further specifying the supergoal, supported by the "unknown" portion of the supergoal content at any given point - shapers are not are not meta-supergoals.
      If the super-supergoal is producing supergoal definitions that maximally satisfy humans, then the system can short-circuit by directly modifying human brain states for maximal satisfaction (for example).  If a shaper stated that supergoals should maximally satisfy humans - this isn't a real shaper, though it's a good heuristic - then the current supergoal content about volitional consent can still veto the idea of nonconsensually modifying the brain state of the programmer, because shapers aren't supergoals and don't have the power to contradict supergoals.  Shapers define supergoals.  "Shaper-pondering behaviors" are subgoals about how to build the supergoals; they may be behaviors that affect supergoals, or rather the current description of the supergoals, but such behaviors still can't override supergoals, no more than any other subgoal can.  The AI's task of modeling shapers and modifying supergoals accordingly is supported by the currently unknown supergoal content, and can't randomly stomp other supergoal content in the same way that a "meta-supergoal" could stomp supergoals.  (Note:  This is only a rough explanation.)
      An "anchor" is a feature of AI philosophy that reflects the fact that the programmers themselves may not understand their own tangled goal system.  An "anchor" is a probable output of the shaper system whose causal antecedents are not fully known; an "anchor" happens when a human says "I think X is a good idea", but isn't clear about why, or isn't sure he/she is fully self-aware of all the valid factors that played a part in the decision.  An anchor is an external reference to whichever causes were involved in the decision; it refers to whichever factual cognitive events were occurring within the human's brain.  An anchor is a pointer for the AI to learn, by examination of the humans, about a portion of the shaper system that the humans themselves are fuzzy about.  See anchor.
• Singularity: 
      The transition in human history marked by the technological creation of greater-than-human intelligence; the end of the era of "strictly human intelligence" that has embraced the last fifty thousand years.  If one were to draw a single line, dividing into two parts the history of Earth-originating intelligent life, the line would be drawn at the Singularity.  See "An Introduction to the Singularity".
• Singularity-safed:  Defined in 3.5: Developmental Friendliness.
      A system such that the immediate start of a hard takeoff does not signal automatic catastrophe.  Broadly speaking, there are three classes of Singularity-safed AIs:  (1) A mature AI with structurally complete Friendliness semantics and the threshold level of Friendliness content, such that the acquisition of transhuman intelligence will probably enable the AI to complete the Friendship system with no further inputs.  (2) A young AI with an ethical injunction telling ver to slow down and ask for directions in case of a hard takeoff.  (3) Any recursively self-improving AI, no matter how primitive, that has a programmatic "controlled ascent" feature which triggers a save-and-stasis if improvement suddenly accelerates.  See 3.5.3: Singularity-safing ("In case of Singularity, break glass").
• subgoal:  Defined in An Introduction to Goal Systems.
      An event or world-state whose desirability is contingent on its predicted outcome.  An action taken in order to achieve some other goal.  A thing not desirable in itself, but rather desirable for some other end.
      If the "parent goal" is a wet sponge, then placing a dry sponge in water is a "child goal" of getting a wet sponge.  The parent goal, in turn, is probably a child goal of something else.  "Parent goal" and "child goal" describes the relation between two individual goals, while "supergoal" and "subgoal" describes the general distinction between things intrinsically desirable, and things which are desirable only because they are predicted to lead to other things.
      It is very important to distinguish between supergoals and subgoals in Friendly AI, since subgoals are directly and entirely dependent on the current world-state.
• subgoal stomp:  Defined in 3.2: Generic goal systems.
      A "failure of Friendliness" scenario in which a subgoal stomps on a supergoal - for example, putting on your shoes before your socks, or turning all the matter in the Universe into computronium because some (ex-)petitioner asked you to solve the Riemann Hypothesis.
• subjective: 
      Existing within the mind, as opposed to external reality.
      In the strict sense, virtually all things except quarks are subjective.  See objective.
      In the nonabsolute sense, "subjective" is often used to indicate a statement which is subject to observer bias or speaker deixis, or which makes visible reference to structures that exist only within the mind.
• supergoal:  Defined in An Introduction to Goal Systems.
      Supergoal content determines which states of the world are seen as intrinsically desirable.  Opposed to "subgoal", a state desirable only as a means to an end.  (The terms "parent goal" and "child goal" are used to refer to local relations between goals.  Goal A may be a child goal of B, which in turn is a child goal of C, making B the "parent goal" of A and the "child goal" of C.  By contrast, the distinction between supergoal and subgoal is a distinction in kind, at least under the CFAI architecture.)  See also subgoal.
• superintelligence: 
      A level of ability enormously in advance of that possessed by humans.  Despite common usage, there is probably no such thing as a "superintelligent AI"; a true superintelligence would no more resemble an AI than a human resembles an amoeba.  Singularity analysis commonly distinguishes between weak superintelligence, which is human thought at enormously higher speeds, and strong superintelligence, involving an intelligence difference similar to that distinguishing a human from a chimpanzee, or a dog, or a possibly a rock.
      It has been suggested (6) that "transhuman" should refer to a mind with perhaps three or four times the computing capacity (or cognitive capacity) of a human, that "superintelligence" should refer to a mind with perhaps a hundred or a thousand times human capacity, and that "Power" (a la Vinge) should refer to a mind with billions or trillions of times the computing power of Earth's entire current population.  It is noteworthy that even a transhuman is still smarter than we are and thus fundamentally incomprehensible, and also that a desktop rod logic suffices for superintelligence under this definition.
• supersaturated Friendliness:  Defined in 3.5: Developmental Friendliness.
      A Friendly AI that has as much Friendliness structure and Friendliness content as ve can usefully represent (for use then or later) is supersaturatedly Friendly.  Supersaturated Friendliness is achieved by adding ethical injunctions long before they're needed, letting the AI avoid anything that looks the least bit unFriendly "just to be sure", trying to implement the last 10% of functionality that takes 90% of the effort, and doing it all two or more steps in advance of the development stage where it becomes unavoidably necessary.
• Sysop:  Defined in 1: Challenges of Friendly AI.
      A superintelligent AI which provides an "operating system" for all the matter in the Solar System.  A living peace treaty with the power to enforce itself.  A friendly superintelligence ready to do whatever the "inhabitants" ask - including upgrading us to superintelligence ourselves - so long as it doesn't violate the citizenship rules derived from Friendliness.  (For example, asking the Sysop to kill someone else, or to use more than your share of the Solar System's resources, or even to upload and upgrade someone against their will.)  One of the possible outcomes of the Singularity; see Interlude: Of Transition Guides and Sysops.
• transhuman: 
      A level of ability substantially in advance of that possessed by humans.  A transhuman AI is one which has substantially exceeded human intelligence.  A further distinction can be made between weak transhumanity, which involves intelligence differentials roughly of the same order as those that distinguish Einstein from the average human, and strong transhumanity, involving an intelligence difference similar to that distinguishing a human from a chimpanzee.  See also the definitions of weak and strong superintelligence.
• Transition Guide:  Defined in 1: Challenges of Friendly AI.
      For the last fifty thousand years, humanity has been essentially at the mercy of a hostile universe.  Post-Singularity, things will be different.  A Transition Guide is a Friendly seed AI that has achieved superintelligence and created nanotechnology, or whatever other tools are needed to (a) perceive whatever state of existence comes after the Singularity and (b) handle the transition.  A Transition Guide is the bridge between modern-day humanity and the other side of a Friendly Singularity.
• truly perfect Friendly AI:  Defined in 1: Challenges of Friendly AI.
      An AI which always takes the perfectly Friendly action, even if you programmed it to do something else.  A truly perfect Friendly AI has sufficient "strength of philosophical personality" - while still matching the intuitive aspects of friendliness, such as not killing off humans and so on - that we are more inclined to trust the philosophy of the Friendly AI, than the philosophy of the original programmers.  The real-world analogue would be a "philosophically human-equivalent" or "philosophically transhuman" Friendly AI.
• unFriendly: 
      An unFriendly action: an action which violates Friendliness.  An unFriendly AI: an AI which is not designed using Friendly AI design principles, or an AI which has undergone a catastrophic failure of Friendliness.  Intuitively:  The AI starts shooting people (if near-human) or exterminates humanity and dissassembles us for spare atoms (if transhuman).
• unity of will:  Defined in 3.3: Seed AI goal systems.
      See moral deixis and 3.3.3: Unity of will.  The ability of Friendship programmers and AIs to cooperate, even against the AI's "interests" (from a human perspective, at least), because their wills and decisions are sufficiently close in the near-term, and sufficiently convergent in the long-term.
      Given perfect identity of knowledge, supergoals, and cognitive processes, the result is perfect unity of will.  Small variances in knowledge resulting from different sensory viewpoints should not be sufficient to break the characteristic functionality of "unity of will", unless the variance results in a desirability differential greater than the differential desirability of the "unity of will" process.  See 3.3.3: Unity of will for more details.
• upload: 
      Colloquially, a human being that has been uploaded into a computer is an "upload".  The classic method is a "Moravec Transfer"; a medical nanobot enters the brain, swims up to a neuron, scans the neuron until the outputs can be perfectly predicted (or predicted to within an order of magnitude greater accuracy than the neuron's sensitivity to random external thermal fluctuations), replaces the neuron (slowly or quickly), and takes over that neuron's functional role within the brain - perhaps first enclosing the axons, dendrites, and cell body in a robotic casing, then switching from the cell body's outputs to the nanobot's computed outputs.  If this procedure is repeated for each neuron in the brain, a mind can actually be transferred from biological to computational substrate without ever losing consciousness.  Presumably hormones, glands, neurotransmitter diffusion, electrical field effects, long-term potentiation, and so on, would need to be simulated as well.  But a neuron is a lot larger than a medical nanobot, so there's plenty of room to scan right down to the individual microtubule dimers.
      If human neurons are noncomputable a la Hameroff and Penrose, then the nanobots and the new computational substrate need to employ noncomputable hardware as well.  In essence, the mind would be transferred to programmable high-speed "superneurons" rather than silicon transistors as such.
• wirehead:  Defined in 2: Beyond anthropomorphism.
       Larry Niven once wrote a story, "Death by Ecstasy", dealing with the possibility of humans wiring their pleasure centers with electrodes - a habit he called "wireheading".  Another story that deals with the issue of wireheading is the novel Mindkiller by Spider Robinson.  Hence the term.
      In Friendly AI terms, a "wirehead AI" is one that has substituted the indicator of success or desirability for the supergoal content itself.  The "wirehead fallacy" is the idea that any sufficiently intelligent entity necessarily values pleasure above all else.  The "wirehead problem" is any failure of Friendliness in which an internal indicator becomes the de facto supergoal; "wirehead problem" also refers to any failure of the goal architecture in which some piece of mindstuff "cheats" and assigns itself desirability (this problem turned up fifteen years ago in Doug Lenat's Eurisko; see 3.3.5: FoF:  Wireheading 2).
• wisdom tournament:  Defined in 3.3: Seed AI goal systems.
      One of the major sources of human wisdom is our need to compensate for our built-in errors.  We invented science to deal with politically biased belief systems.  We invented altruism to deal with our selfishness.  We invented ethics to deal with our social observer-biased rationalizations.  A wisdom tournament is a subjunctive, self-simulated version of the AI that shows the AI what the AI "would have done" if the AI had been dumber, slower, biased.  Without introducing actual errors, the AI gains reliable information about what would have happened if the AI had been running on unreliable simulated hardware, with random perturbations to the software, ethical misinformation, factual misinformation, tempting ends-justify-the-means scenarios, and an instinct to kill and destroy.  If the Friendliness, and the heuristics the AI has learned from past tournaments, are enough to walk through all that without making a single major error, then the Friendliness may be strong enough to face real life.
      See 3.3.4: Wisdom tournaments.

• 1:  The meta-inhibitions can protect themselves, though, so the recursion stops there.  There may be an infinite number of avenues to be inhibited, but a single inhibition doesn't necessarily involve infinite recursion.
• 2:  It is possible that mathematics is objective.  It is also possible that the mathematical laws we observe are merely derivative patterns; higher-level behaviors exhibited by the true laws of physics.
• 3:  How fast were you going when you made the measurement?
• 4:  Another Special-Relativity thing.  If you run past the ruler at 75% of the speed of light, measuring the ruler will show that it's around half the length you would otherwise expect.
• 5:  At least, the local speed of light.  I'm not getting into General Relativity.
• 6:  I cannot recall who originally made the suggestion.

Next:	A.4: Version History
Up:	Appendix A: Friendly AI Guides and References
Prev:	A.2: Complete Table of Contents