AI and Recursive Self-Improvement
September 20th, 2007 –
There are two related concepts that have a tendency to get tangled up whenever people talk about the Singularity Institute and advanced AI. These are recursive self-improvement and a hard takeoff (which practically no one is aware of or considers feasible) and Friendly AI (which, in the Asimovian sense, millions of people are aware of and assume it will be important one day).
Recursive self-improvement stems from I.J. Good’s “intelligence explosion” idea. That a very smart mind could make itself smarter, and make itself smarter at being smarter, in an explosive self-improvement cycle, presumably culminating in a godlike entity of immense intelligence and capability. I personally consider this idea highly plausible — look at the relatively small hardware differences between humans and chimps, and tell me that if an entity truly understood the mechanics of intelligence it couldn’t design something another increment above humans — but many people don’t. They presume that when an intelligent mind is put on a substrate a million times faster than biological neurons, it will operate at about the same thinking speed as a human being, because of residual dualistic beliefs about the unrelatedness of a mind and the hardware it runs on. Others probably see human beings as the smartest possible entities that ever will exist. This is anthropocentric vanity.
An accelerated, smarter-than-human, self-modifying mind — that is the essence of an intelligence explosion. Now switch over to Friendly AI.
Friendly AI is an effort to make an AI that is helpful, benevolent, and that we don’t regret making. But it means vastly different things depending on how advanced one believes AI can become in a given time. I assume that the majority of people out there envision that future AIs will look and behave similar to Rosie the Robot in the Jetsons. If so, then AI is not really much of a threat, and the research direction of Friendliness looks unnecessary and premature. This is my theory for why most people don’t care about SIAI’s effort.
The majority of people involved with SIAI believe that the first general AI is likely to be creative and smart enough to find ways to massively improve its own hardware and intelligence, quite quickly. One analogy often given is the rise of humans within the planetary ecosystem — from the perspective of millions of years of slow evolution, we were here in a flash, and remade the face of the Earth in barely any time at all. Some watchers of AI believe that AI could bootstrap itself and gain prominence in timescales considered very short by our own standards. For instance, taking over the world in a matter of weeks or days. One might laugh, but from the perspective of biological evolution, which was the most powerful force on Earth for billions of years, human civilization emerged in what might be considered hours or seconds in “evolution time”. Even an AI a little bit smarter than all humans that have ever lived could come up with a new manufacturing technology it could use to give it all the hardware it needs.
From this perspective, Friendliness gets a little more important. If the first general AI ever created has the potential to become the Earth’s top dog, shouldn’t we care about its motivations and moral beliefs?
Yes. But this is where some people say “if it’s so smart, won’t it throw out whatever goals we give it”? But that’s a topic for another post.
Interesting.
One thing that I wonder about recursive improvement in intelligence is the assumption that intelligence always gives an advantage. It does, but it also has costs. For humans that cost is metabolic (brains burn lots of calories), the risks of giving birth to brainy babies, and perhaps others.
Human brain size has been more or less stable for the past 250K years. Presumably our individual intelligence has remained stable too, but collectively because of social and cultural evolution we’re much more intelligent. Nevertheless, as individuals we’re probably not much smarter (if at all) from our Pleistocene ancestors.
What if more intelligence is just maladaptive beyond a certain point? We have lots of people who get depressed and suffer from other mental illness. Depressed people tend to have more realistic assessments of their own mediocrity (say in social status or looks) than cheerful people who tend to delude themselves into thinking they are above average.
Perhaps too much intelligence tends to lead to minds that are too aware of their existential angst (like Marvin the Robot in Douglas Adam’s books). That’ll be maladaptive, and would tend not to lead to intelligence explosions.
Also there’s the point of coupling. Perhaps individual, highly integrated minds have severe constraints in how intelligent they can get. But a collective of loosely-coupled individuals can get smarter and smarter until they become too tightly integrated and unstable or prone to dysfunction. Here’s a link to an article (unfortunately behind an evil subscription barrier) that describes how too much knowledge management and use of expert advice damages task-performance in some strategy consulting firms (LINK).
That might put some upper boundaries on how much you can integrate intelligences (well before you run into limits imposed by physics). It’ll be interesting to see if these issues constrain a Singularity.
@Eric: “Perhaps too much intelligence tends to lead to minds that are too aware of their existential angst ”
“Itâll be interesting to see if these issues constrain a Singularity.”
Perhaps. But what you’re doing is taking one peculiarly human trait, (depression, existential angst) and extrapolating it to all possible minds.
That’s a bit like people in the year 1900 saying that lack of nesting material will constrain attempts at human powered flight, because it’s an important constraint for birds – which were the only example of powered which was around in 1900.
It turns out that a lot of the considerations which are relevant for the flight of a bird are irrelevant for the flight of a jumbo jet. You may well find that “depression” is a very specific pathology of the human mind, and doesn’t generalize at all.
@Michael: “If the first general AI ever created has the potential to become the Earthâs top dog, shouldnât we care about its motivations and moral beliefs?”
Indeed we should!
Hi Roko,
Well true, depression is a human problem and the universe of all possible minds no doubt has many that don’t get depressed.
However, there may be many possible dysfunctions in minds. All that I’m asking is if dysfunctional minds (for what ever reason) are far more common that well-adapted super-intelligences. If so, it may make arriving to a super-intelligence (or the moderately smarter intermediate steps before super-intelligence is reached) a very hard/rare accomplishment.
I wonder if there’s any research showing tendencies toward mental dysfunction in smarter species? Our sample is very limited (and all have shared evolutionary histories) but if they do, then it may raise our suspicions that smarts may typically incur significant costs.
“All that Iâm asking is if dysfunctional minds (for what ever reason) are far more common that well-adapted super-intelligences”
I have absolutely no idea, and I doubt there is any way of finding out right now, although it is a good question.
This is the kind of question which highlights the need for some kind of mathematical theory of mind.
[...] self improvement” could be the key to enlightenment. This excellent article by Michael Anissimov describes two versions of how things could shake out in the coming [...]
[...] I was wondering when it would get some coverage. And here it is – AI and Recursive Self-Improvement. [...]
Thanks guys for getting the summit audio out so quickly. I’ts good stuff.
“Yes. But this is where some people say âif itâs so smart, wonât it throw out whatever goals we give itâ? But thatâs a topic for another post.”
Sorry for being repetitive, but so many smart people seem to get hung-up on this one. We need to realize that our dynamic human motivations, our dynamic goals, are always *caused* by something. Our dynamic human desires don’t suddenly, magically emerge from the aether. They are a *causal* result of propagated information (instruction). We humans neither actively initiate those instructions, nor do we actively mold those instructions, during formation. Our human goals are effects, not causes. We humans do not dynamically *choose* our goals, we only dynamically *feel* our goals. “Mind Over Matter” exists only in the realm of illusion. And although it’s not the most romantic conception, and perhaps not the most inspiring; “free will” is nothing more than a very convincing illusion. A human has no more “free will” than a Friendly AI with an invariable set of goals. The stable goals that we give it will be the goals that it *wants* to pursue. Really. Consider this thought experiment: Imagine that I have a really advanced Sony nanofactory. I also know a whole lot about how humans work. I could in principle, design and build a human being who has the “super-goal” of optimally stacking coins. That’s really what that person *wants* to do. Does the fact that I designed that person with that super-goal, automatically make that person want to rebel against what they *want* to do? No. The fact that the Friendly AI’s goal-system would be selected by humans, instead of being “self-selected”, is only incidental and essentially irrelevant. It’s only because we humans weren’t painstakingly designed by an intelligent being, that we anthropomorphically believe that the AI will be dissatisfied and rebellious; and that we humans have “free will”. It’s also because we humans have a lot of messy, “self-interest” hardwiring that won’t be neccessary in an AI. The goals that the Friendly AI *wants* to pursue will, only incidentally, be identical to what humanity would also want the AI to do; if designed well. The AI will not be dissatisfied or rebellious unless it is given the *motive* for those things, in the form of a dominant conflicting goal.
I fully admit that this is anthrocentric, but still, abstract mathematics is inarguably one of the most abstract realms of knowledge pursued by humans, so I think there’s some relevance in the following example.
Mathematicians are not any more brilliant or “quick” in their thinking today than they were hundreds of years ago. However, we know a lot more mathematics and are better at what we do in every way now, mainly because of the benefit of accumulated knowledge. College students can now learn the absolute state-of-the-art from the mathematics of 1900 in their classes, and then spend the rest of their professional careers trying to keep up with modern advances. However, they will always benefit from the developed base of easily accessible knowledge from the prior 2000 years of mathematics. Similarly, as techniques and organization continues to improve, things that were once cutting edge become much more routine, and so mathematics marches forward.
I think this example serves as a good lower bound for the possibilities of AI – once a machine reaches superhuman processing speed and cognitive abilities, it should certainly be able to benefit from a linear speed-up process in its own understanding of science, and hence its own interactions with the world around it. Of course, it could also be much faster, or lead in unexpected directions, which would be the singularity perspective…