I tend to believe that you can’t fully separate intelligence and consiousness even if the conciousness or ‘self awareness’ is rudimentary. When you look at the human brain, there’s a basic operating system of instincts. These instincts (or algoritms if you wish) are built in with a fairly full compliment. Like DNA much is preprogrammed and also like DNA, there are slight variations. In addition to those variations, experience, in the form of both testing and sensory input, will either strengthen, weaken or obliterate some of the algoritms. This is learning on a human level where decision theory comes into play in part. What dicision theory needs to include is the temporal continuum of new information that may be relevant or not. That new information changes with the [random] variations of daily existence.
For example: Excess heat placed close to an extremity causes an unconscious or instinctual reaction that is essentially preprogrammed. The learning to recognize and store the information in terms of both a variety of sensory data (visual, environmental, sound, tactile, and taste) and coorelating that in a temporal construct, creates a specific data set that is then stored, repeated instances of this action both strengthen specific algorithms as well as aspects of the data in the construct and also delete or minimize other data so that a predictable action can be planned prior to being hurt again. What was the most obvious associated attribute of the experience besides the pain? What was seen (color, shape, pattern) immediately before it. What was heared, smelled, felt? How close to the contact with the heated surface were the other experiences.

I believe that intelligence is not merely predicting the action, for that would be merely a simpler equation. It requires the ability to make the assesment and to learn what parts of the data are significant and which are not.

When an assumption that a specific variable may always be present that is irrelevant to the process, the learning will be flawed, but it will still be intelligence.

The issue of sensory or data input is necessary but not the basis for intelligence. It is the ability to choose, concsiously, which data to ignore and which to give attention to, overriding the basic instinctual reaction.

Good AI vs. bad AI is not possible with algorithms alone. With AGI, this seemingly random aspect must be introduced. Simple trial and error is insufficient unless your only addressing AI and not AGI. A trial and error approach will suffice for Ben F Rayfield’s description, but Michael Anissimov’s own posts and writings on the subject have already addressed some of these issues.

The exact definition of intelligence is a game between 2 players/symbols X and Y: Its like rock-paper-scissors but with 2 choices instead of 3. Each player chooses X or Y then sees what the other player chose. If both choices equal, player X gets 1 point, else player Y gets the point. After many rounds, the player with the most points is more intelligent because this is the simplest form of pattern-matching and prediction.

Solomonoff Induction is based on this idea, and it helps me build better AI.
http://en.wikipedia.org/wiki/Solomonoff_induction

I’ll put it into context of Newcomb’s Paradox of decision theory.
http://en.wikipedia.org/wiki/Newcomb%27s_paradox

If, for example, you are playing a game where you choose X or Y repeatedly (like the game above) and every time, you lose, because the other player appears to know what you were going to choose before you chose it, then there are 2 probable things to consider:
(1) The other player is much smarter than you, or
(2) The other player can see the future.

I’m not a physics person and I’ve never seen any proof that you can’t see the future or change the past or build a time-machine or any number of unusual things. I don’t know, and I won’t assume, therefore I must consider (1) and (2) above. The same way as you learn to choose X and Y, you can learn to choose (1) or (2) as the most probable theory.

That is how I define intelligence. As an expert on artificial intelligence, I say the X and Y game is the exact definition of intelligence, but that still leaves us the task of putting it into context in our everyday reality.

Perhaps the problem isn’t the classical math, but the assumptions about what is being processed. Perhaps human Reflection also involves an “infinite” recursion, among other factors. (The human brain has *some sort* of throughput flowing through it all the time, anyway). And where the recursive throughput is always *changing*, it would indeed just create an endless (and effectively useless) stream of conceived: a sub-goal of a sub-goal of a sub-goal of a sub-goal… ad infinitum. And nothing would ever get accomplished. But what if the “main goal” or what I’ll call the 1st-Tier goal was “preserved” from continuous modification and was repeatedly “released” into the throughput stream? (For example, preserved algorithmically – by being stored in short-term memory like humans seem to use. Or being preserved as a separate data-file that’s fed into the algorithm). For every cycle of the preserved 1st-Tier goal, the algorithm would conceive/generate a (distinct) 2nd-Tier sub-goal. (Distinct perhaps because human cognitive algorithms are constantly changing ever so slightly in response to processing throughputs. Perhaps this effect could be reproduced by virtue of the AI’s learning sub-routine… Or perhaps the 2nd-Tier sub-goals are individually distinct for a different reason). For example, after 50 cycles of the 1st-Tier goal, 50 (distinct) 2nd-Tier sub-goals will be conceived/generated; as well as 50 (distinct)3rd-Tier sub-goals and so on. In the “background” all of the tiered goals are also still being processed in the “infinite” recursion. For example, a cycle of the 1st-Tier goal generates a 2nd-Tier sub-goal, a cycle of the 2nd-Tier sub-goal generates a 3rd-Tier sub-goal… towards generating an Nth-Tier sub-goal. But the important point is that the preserved (unmodified) 1st-Tier goal is repeatedly being released as throughput through the algorithm (along with all the others). Perhaps attention-allocation accounts for the remainder of the phenomenon. Perhaps our human attention allocation remains “focused” “near the top” where our thinking is dominated by the consideration of perhaps mostly the 1st, 2nd, and 3rd-Tier sub-goals and very little attention is allocated to the more distantly generated sub-goals like the 99th-Tier sub-goal. So mostly what we are (repeatedly) thinking about is 2nd and 3rd-Tier subgoals (just to pick out a small number). Perhaps human Reflection is the result of a complex interplay between attention-allocation, a “preserved” 1st-Tier goal (a super-goal) implemented as repeated throughput, and an “infinte recursion” algorithm (eg. the human knowledge matrix). Anyway, I just made this up from ~0 knowledge of Decision Theory, so it might not be useful, but whatever… ‘Till next time.

I meant to say: No throughput – no thoughts, and no goal perception or satisfaction.

To add a bit more. By pre-specifying the intial throughputs (in the form of something the baby AGI has already learned something about – eg. “Bring me a cup of coffee.”) and tightly controlling the throughputs while being propagated through the algorithms; one might be able to reliably assign the Friendly goals in that fashion. The Friendly throughput would need to be in a “recognizable” form to start with; but it doesn’t necessarily need to be pre-processed by a sensory system – like human brains include. It could take the form of a *pre-specified* “thought” if you will. The baby AGI doesn’t necessarily need to *hear* or *see* the words : “Bring me a cup of coffee.” Although one could also have it pre-processed too, if desired. Or so I say.

“try everything, and keep doing something if your human master likes it””.

If it uses algorithms then the resulting AGI will have to be goal-driven. The goals being (either directly or indirectly) probabalistically derived from the low-level procedures; where the procedures are reflected in the “constructed” throughputs. No throughput – no thoughts and no goal satisfaction. That’s what I’m guessing at least. The possiblity that humanity will select the specific goals is irrelevant from the AGI’s perspective. In fact, it’s physically impossible to even *create* an AGI *without* intentionally or unintentionally, selecting a set of (possibly basically random) initial goals.

“This is no functional AGI design.”

Nobody has claimed that AIXI (as it currently stands) is a practically functional AGI. It’s more of a technical specification of what an *optimal* AGI would be. But I think that it’s an invaluable step in the correct direction. Until AIXI, nobody had been able to provide a universal mathematical formalization of what an AGI even is, precisely.

“Not that I think that AGI is an insoluble problem – far from it, but I just think that AIXI is a bit of a dead end.”

I disagree. But that’s what keeps our world turning.

I dunno about that. I’ve read about the AIXI theory, and it basically boils down to

“try everything, and keep doing something if your human master likes it”

This is no functional AGI design.

Not that I think that AGI is an insoluble problem – far from it, but I just think that AIXI is a bit of a dead end.

After all, when we humans seek to accomplish a multiple-part goal, it is the (dynamic) higher-level “super-goal(s)” that we store in short-term memory, while we focus on first achieving the necessary sub-goals (as part of that particular super-goal). Perhaps favorably weighting the attention allocation would be another useful strategy, to help with Friendliness.

I’m going to hazard a guess and make a proclamation. If I turn out to be wrong, someone can tell me “I told you so!” a billion years from now. I think that any functional AGI design will have to be based on algorithmic probability. Check out the “Gentle (?) Introduction” to AIXI by Marcus Hutter. It’s a mathematical formalization of AGI (but an actually computable version of AIXI hasn’t yet been devised). But I think it’s clear that progress is being made in that direction. I think it will be, *at most*, 20 years before AGI is solved (I would guess, closer to 10). Let’s have Friendliness theory complete before that time.