The Singularity Institute Blog

I was amused and delighted to read the following rather transhumanistic article in the New York Times recently.

http://www.nytimes.com/2007/10/26/opinion/26brooks.html?_r=1&oref=slogin

The writer, who does not appear to be a futurist or transhumanist or Singularitarian or anything like that, is observing the extent to which he has lost his autonomy and outsourced a variety of his cognitive functions to various devices with which he interacts. And he feels he has become stronger rather than weaker because of this — and not any less of an individual.

This ties in deeply with the theme of the Global Brain

http://pespmc1.vub.ac.be/SUPORGLI.html

which is a concept dear to my heart … I wrote about it extensively in my 2001 book “Creating Internet Intelligence” and (together with Francis Heylighen) co-organized the 2001 Global Brain 0 workshop in Brussels.

I have had similar thoughts to the above New York Times article many times recently… I can feel myself subjectively becoming far more part of the Global Brain than I was even 5 years ago, let alone 10…

As a prosaic example: Via making extensive use of task lists as described in the “Getting Things Done” methodology

http://en.wikipedia.org/wiki/Getting_Things_Done

I’ve externalized much of my medium-term memory about my work-life.

And via using Google Calendar extensively I have externalized my long-term memory… I use the calendar not only to record events but also to record information about what I should think about in the future (e.g. “Dec. 10 — you should have time to start thinking about systems theory in connection to developmental psychology again…”)

And, so much of my scientific work these days consists of reading little snippets of things that my colleagues on the Novamente project (or other intellectual collaborators) wrote, and then responding to them…. It’s not that common these days that I undertake a large project myself, because I can always think of someone to collaborate with, and then the project becomes in significant part a matter of online back-and-forth….

And the process of doing computer science research is so different now than it was a decade or two ago, due to the ready availability and easy findability of so many research ideas, algorithms, code snippets etc. produced by other people.

Does this mean that I’m no longer an individual? It’s certainly different than if I were sitting on a mountain for 10 years with my eagle and my lion like Nietzsche’s Zarathustra.

But yet I don’t feel like I’ve lost my distinctiveness and become somehow homogenized –
the way I interface with the synergetic network of machines and people is unique in complexly patterned ways, and constitutes my individuality.

Just as a neuron in the brain does not particularly manifest its individuality any less than a neuron floating by itself in a solution. In fact, the neuron in the brain may manifest its
individuality more greatly, due to having a richer, more complex variety of stimuli to which it may respond individually.

None of these observations are at all surprising from a Global Brain theory perspective. But, they’re significant as real-time, subjectively-perceived and objectively-observed inklings of the accelerating emergence of a more and more powerful and coordinated Global Brain, of which we are parts.

And I think this ties in with Ray Kurzweil’s point that by the time we have human-level AGI, it may not be “us versus them”, it may be a case where it’s impossible to draw the line between us and them…

– Ben

P.S.

As a post-script, I think it’s interesting to tie this Global Brain meme in with the possibility of a “controlled ascent” approach to the Singularity and the advent of the transhuman condition.

Looking forward to the stage at which we’ve created human-leve AGI’s — if these AGI’s become smarter and smarter at an intentionally-controlled rate (say a factor of 1.2 per year, just to throw a number out there), and if humans are intimately interlinked with these AGI’s in a Global Brain like fashion (as does seem to be occurring, at an accelerating rate), then we have a quite interesting scenario.

Of course I realize that guaranteeing this sort of controlled ascent is a hard problem. And I realize there are ethical issues involved in making sure a controlled ascent like this respects the rights of individuals who choose not to ascend at all. And I realize that those who want to ascend faster may get irritated at the slow pace. All these points need addressing in great detail by an informed and intelligent and relevantly educated community, but they aren’t my point right now — my point in this postcript is the synergetic interrelation of the Global Brain meme with the controlled-ascent meme.

The synergy here is that as the global brain gets smarter and smarter, and we get more and more richly integrated into it, and the AGI’s that will increasingly drive the development of the global brain get smarter and smarter — there is a possibility that we will become more and more richly integrated with a greater whole, while at the same time having greater capability to exercise our uniqueness and individually.

O Brave New Meta-mind, etc. etc.

Marvin Minsky in an interview with Danielle Egan for New Scientist:

Minsky: The reason we have politicians is to prevent bad things from happening. It doesn’t make sense to ask a scientist to worry about the bad effects of their discoveries, because they’re no better at that than anyone else. Scientists are not particularly good at social policy.

Egan: But shouldn’t they have an ethical responsibility for their inventions?

Minsky: No they shouldn’t have an ethical responsibility for their inventions. They should be able to do what they want. You shouldn’t have to ask them to have the same values as other people. Because then you won’t get them. They’ll make stupid decisions and not work on important things, because they see possible dangers. What you need is a separation of powers. It doesn’t make any sense to have the same person do both.

The Singularity Institute was recently asked to comment on this interview – which by the time it made it through the editors at New Scientist, contained just the unvarnished quote “Scientists shouldn’t have an ethical responsibility for their inventions. They should be able to do what they want. You shouldn’t have to ask them to have the same values as other people.” Nice one, New Scientist. Thanks to Egan for providing the original interview text.

This makes an interesting contrast with what I said in my “Cognitive biases” chapter for Bostrom’s Global Catastrophic Risks:

Someone on the physics-disaster committee should know what the term “existential risk” means; should possess whatever skills the field of existential risk management has accumulated or borrowed. For maximum safety, that person should also be a physicist. The domain-specific expertise and the expertise pertaining to existential risks should combine in one person. I am skeptical that a scholar of heuristics and biases, unable to read physics equations, could check the work of physicists who knew nothing of heuristics and biases.

Should ethicists be inside or outside a profession?

Read the rest of this entry »

New additions to the Singularity Institute’s team:

Pejman Makhfi, SIAI Director of Venture Development

Pejman Makhfi engages with investment communities on behalf of SIAI, and advises researchers and entrepreneurs to help them develop new ventures. He is a Silicon Valley technology veteran, entrepreneur, and investor, with fifteen years of experience consulting entrepreneurs, technology investors, and forward-thinking startups. He is founder and managing director of Venture Choice, a private angel group. He is widely recognized as a leader in the field of business process automation and knowledge modeling, and served as the key architect for several award-winning leaders in the software and financial industries, including FinancialCircuit and Savvion. Pejman holds a B.S./M.S. in Computer Science from the University of Dortmund in Germany. He has authored multiple patents and standards, and is a contributor to IEEE Computational Intelligence Society, the Association for the Advancement of Artificial Intelligence, and American Society for Quality.

Email: pejman(at)singinst(dot)org

C. Colby Thomson, SIAI Associate Director of Strategy

C. Colby Thomson focuses on strategy, messaging, public relations, and outreach for SIAI. He is CEO of Allied Strategy, a small, innovative, software company serving the insurance industry. His primary areas of interest include financial services, health care, globalization, AI, nanotechnology, and other futures topics. He received his BS in Computer Science from the University of Lincoln-Nebraska in 2004, and his MBA in Business Administration in 2005 from UNL’s J.D. Edwards Honors Program in Computer Science and Management. He actively supports entrepreneurship in the Lincoln, Omaha, and Greater Kansas City areas, serving as a board member of Turbine Flats and the Lincoln Young Professionals Group, chairman of the Kansas City Council of Young Entrepreneurs, and Tri-Chair of the 2015 Vision Group: University R&D Corridor.

Email: thomson(at)singinst(dot)org

I was once present when William Hurlbut, during a debate with Aubrey de Grey, spoke of “the meaning that death gives to life”; Hurlbut repeated the standard claims that life without death would be meaningless and empty. As I replied during the comments session, Hurlbut had not made a sincere effort to think about what meaning immortality would give to life, on the same order of the effort that has gone into thinking about “the meaning that death gives to life”.

Philosophers have put forth a mighty effort to find nice things to say about death. But this is scant reason to fear lifespan extension, when philosophers have not put forth an equally motivated effort to say nice things about immortality.
Read the rest of this entry »

Excited by the X-prize and the Netflix Challenge and so forth, a number of people have asked me recently about the possibility of making some sort of “prize” for Artificial General Intelligence.

It took me a while to come up with any idea that made sense, but I think I finally have one. (It’s not all that original, it’s basically just a variation of a well-known idea … but hey … although I’m probably the world’s biggest fan of originality, when it comes to something as potentially serious in its implications as an AGI prize, I’d rather be right than wholly original!)

You will likely be familiar with the Turing Test, a test made up by Alan Turing as a potential “sufficient but not necessary” criterion for an AI system to be considered intelligent. Put simply, the Turing test requires an AI to effectively impersonate a human in a text chat with other humans, where the effectiveness of the impersonation is determined by a set of human judges.

The Loebner Prize already exists to reward the first person to create an AI capable of passing the Turing Test –

… but the Loebner Prize has not been at all usefully motivating for the field of AI. Since passing the Turing test appears to require a powerful, human-level AGI, which nobody has at this point, the prize is given each year to whomever fools the greatest percentage of human judges. But, it turns out that, lacking a real human-level AGI, the best way to fool the greatest percentage of human judges that your AI program is a human is to simply write a clever chat bot.

The most interesting Loebner finalist so far, in my view, was Jason Hutchens’ MegaHal system, which turned in a decent performance in the 1998 contest spite of being a wholly statistical learning based system, without a bunch of programmed-in rules. (Hutchens’ less-interesting system, Hex, won the prize in 1996).

So anyway, the Loebner Prize has become a showcase for chatbots rather than for systems seriously attempting to model human cognition or to implement alternate models of cognition.

But, a couple weeks ago when someone asked me once again about prizes for AGI, my mind wandered to the work I’m doing with Novamente LLC involving using our Novamente Cognition Engine to control virtual agents in virtual worlds like Second Life. One of the products we’re considering launching in Novamente, eventually, is a virtual baby for Second Life. Users would subscribe and pay a certain amount each year to own the baby, and it would mature gradually, getting smarter and smarter via its interactions with its parents and the world.

At first I thought of making a kind of Second Life based Turing Test: can your AI control an avatar that acts like a 4-year-old child? Which I do think is interesting. If you teach a 4-year old to use Second Life (which will be easier in a couple weeks when Electric Sheep Company’s new OnRez interface comes out) and have them control an avatar, they will control it in a 4-year-old-ish way … so emulating what a 4-year-old does in Second Life is a decent test of whether you’ve successfully built a virtual AI toddler.

But I didn’t really like the complicatedness of this test. And, it takes an uncommonly clever 4-year-old to use Second Life … at least until OnRez version 3.0 or whatever comes out….

So the next natural step was to remove the virtual world from the test, and just envision a Toddler Turing Test, in which AI’s try to fool human judges into thinking that they are N-year-old children, in the context of a text chat. (Most likely the judges, however, would be human adults rather than toddlers!)

Let’s say, hypothetically, that in 2009 we ran a 4-year-old Toddler Turing Test. If someone passes, then in 2010 we run a 5-year-old Toddler Turing Test. If no one passes, then in 2010 it’s the 4-year-old version again.

[SEE POSTSCRIPT AT THE END: A couple weeks after writing this blog, it was pointed out to me that some folks in China have in fact done a children's Turing Test, and it was a viable experiment....]

In this way, the progress of the Toddler Turing Test can gradually mimic the rate of progress of the AGI technology.

So far I don’t see any major problems with this approach, it seems a viable way to make a prize for AGI research, which would reward actual progress toward AGI rather than development of narrow-AI systems like chat bots.

Note that it may happen that the 4-year-old prize is won by some sort of chat bot. That’s okay. Even so, after a few years a level will be reached where the chat bots totally can’t compete anymore.

One difference I suggest from the Loebner Prize is not to give any prizes to “the best participant each year.” This partial-reward property of the Loebner Prize is much of what makes it so unsatisfying, as it results in prizes (albeit small ones) being given to fundamentally unintelligent chat bots. I suggest that a Toddler Turing Test give an award only to a program that actually passes….

This would also make pretty good theater, if properly video-recorded. Each year, to run the contest, we’d need to get a number of real human toddlers and their supervisors in a room together in front of computers, and have them carry out chats with judges in another room (who would also be conversing with AI’s running on servers sitting in yet another room). As a big fan of human toddlers as well as AGI programs this sounds like a lot of fun to me…

Of course, Toddler Turing Test is not really a good name, since eventually (hopefully before too long) N will get beyond the toddler phase. Probably Developmental Turing Test is better, although it doesn’t have nearly as funkalicious a ring to it….

P.S.

To add some concreteness to the idea, here are some real interviews with 4 and 5 year old humans…

http://www.bloggingawaydebt.com/2006/11/a-very-special-interviewwith-my-4-year-old-son/

http://www.bloggingawaydebt.com/2007/05/a-very-special-personal-finance-interview-6-months-later/

http://paschallstoethiopia.blogspot.com/2007/09/interview-with-4-year-old.html

http://www.youtube.com/watch?v=iZAXiLSn7ts

http://marthabrockenbrough.blogspot.com/2006/05/interview-with-5-year-old-about-gender.html

The 5 year old interviewees are already more advanced conversationalists and cognizers than current AI’s…

P.P.S. — A couple weeks after posting the original version of this blog-post, someone pointed out to me a paper describing an actual Turing Test done with children by some Chinese researchers. Very cool!!

See

Some Cognitive Aspects of a Turing Test for Children

http://www.springerlink.com/content/w7572p14l7r4517n/?p=9c93cf27834c404fbddeab90b2234daf&pi=2

(I can’t find the paper available for free online.)

I read the paper — nothing terribly interesting, but it does provide empirical validation that

• it is quite possible to do a Turing Test type simulation with kids aged as young as 5
• current AI programs (at least the chat bots they tried out in their experiment) can’t pass this Turing test

So it validates that a childrens’ Turing Test starting at age 5 is not a stupid or unworkable idea.

What is intelligence? It’s like pornography — you know it when you see it.

The way people talk, you’d think that intelligence was some sort of magical fairy dust. It’s not. Intelligence is a dynamic system that takes in information about the world, abstracts regularities from that information, stores it in memories, and uses it knowledge about the world to form goals, make plans and implement them.

Simple enough. The definition I give above covers humans, cyborgs, possibly intelligent animals, artificial intelligences, extraterrestrial beings, and whatever other form of intelligence you care to dream up.

There is no intelligence that isn’t a dynamic system. Intelligence is inherently a dynamic process. This one is a no-brainer.

There is no intelligence that doesn’t take in information about the world. It needs that information to learn and make plans. If it isn’t taking in information, something is wrong, like it’s locked in a box. But even an intelligence locked in a box takes in the information that it’s in a box.

There is no intelligence that doesn’t abstract information from its perceptual data. There’s simply too much to store otherwise. It has been estimated that we take in gigabits of perceptual data every second. Only a minority of that makes it into our long-term memory or plays a critical role in concept formation. Unless the data input is artificially constrained, an intelligence will always throw away most of the information it gathers.

There is no intelligence that doesn’t store memories. Even the main character of Memento had a perfectly functioning short-term memory. Without it, intelligence wouldn’t be possible. Lacking memories, we would be completely ignorant of both the past and the future. Not intelligent at all.

There is no intelligence that lacks goals. Even a randomly generated goal is still a goal. Uttering a word, turning your head to look at something, moving aside when a large object is incoming — these are all small goals. Any form of differential desirability constitutes goalhood. Without differential desirability, an intelligence will just sit still until it starves or otherwise runs out of power. An intelligence may derive its goals from external feedback… but that doesn’t mean it doesn’t have goals, just that it copies them from elsewhere.

There is no intelligence that doesn’t make plans. Making plans is an essential part of achieving goals. Intelligences visualize a goal state and then try to come up with a series of actions that will guide the current world into that goal state.

There is no intelligence that doesn’t implement plans. Admittedly, this is the shakiest of the above requirements, as there could exist an intelligence that is an Oracle, making plans but passing them on to others to implement. I would consider that intelligence as implementing its plans indirectly, as making plans inherently implies their possible implementation. Okay, so maybe I cheated on that one.

And so concludes my attempt at coming up with a somewhat detailed nonanthropomorphic definition of intelligence.

It’s not often that I award someone the accolade of having caused a complete system crash in my mind. This year’s award goes to Philosophy Etc. for the following dilemma:

Imagine a universe containing infinitely many immortal people, partitioned into two “spheres”. (Every person is assigned a natural number N.) In one sphere, all the inhabitants live a blissful existence, whereas the members of the other sphere suffer unbearable agony. Now compare the following two variations:

1) Everyone starts off in the blissful sphere. But each day (N), one more person (N) gets permanently transferred across to the agony sphere, where they reside for the rest of eternity.

2) Everyone starts off in the agony sphere. But each day (N), one more person (N) gets permanently transferred across to the blissful sphere, where they reside for the rest of eternity.

Which scenario is better? The answer, paradoxically, appears to be “both”. At any moment in time, there will be infinitely many people in the original sphere, and only a finite number who have been transferred across. So option 1 is better.

However, each particular person will spend only a finite amount of time in the first sphere, whereas they will spend an eternity in their post-transfer home. So option 2 is better.

Read the rest of this entry »

Suppose an evil dictator starts a military research programme to develop an AI strategist. This AI is designed to run military campaigns that will kill thousands of people. Following a series of theoretical insights, the researchers complete the latest generation of AI. This AI is as smart as an exceptional human but runs 1000 times faster. They flip the switch. What happens?

Read the rest of this entry »

What about consciousness? The sense of being one person and not somebody else? The sense of being a captive and concerned audience to the happenings of your particular self?

I think Singularitarians like myself tend to ignore this question — not because it’s unimportant, but because we don’t see it as relevant to our core concerns. Conscious or not, recursively self-improving artificial intelligence presents a risk and opportunity impossible to understate.

And yet, many are concerned that the “higher” human abilities, such as creativity and aesthetic appreciation, are inseparably connected to one’s conscious experience. Artificial Intelligence is insufficient, according to this view, because it could only emulate the types of thinking that are most obviously logical and analytical. To get the rest you would need some sort of Artificial Consciousness, too.

The implication that tends to follow here is that even if AI is possible, AC — Artificial Consciousness — may not be. Rather than jump right in to a rebuttal, however, let me toss this question to the cognitive science experts in the room: To what extent, if any, are human brain functions relating to creativity and aesthetic appreciation linked to those behind our perceived senses of identity and free will? Has the research progressed far enough to tell us?

I’m willing to hazard that an assumption this widespread could be a reflection of an actual relationship in the hardware, at least in the case of humans. So I’m not so quick to dismiss the above reasoning entirely.

But regardless of how research answers that question, we must be wary of assuming that mental capacities are linked in human brains because of properties inherent to intelligence-in-general.

We must remember that our brains are not so different from those of species unable to appreciate a good Picasso. Our newer abilities are owed to existing brain structures recently re-purposed by small evolutionary changes.

Imagine spending your childhood in a home built from an abandoned missile silo. You might have assumed that long ladders are indispensable for connecting large rooms, and that a simultaneous turning of keys by two people on opposite sides of the kitchen is the only reasonable way to start a dishwasher. How many linkages between our intellectual features are similarly silly?

Are some connections truly necessary? I answer with a skeptical maybe. It could well be that any AI capable of love will also have a kind of consciousness. But at this point in time I don’t know how to test that assumption. And apart from the obvious philosophical questions this raises, I’m still not convinced it matters.

As I was recently telling a colleague, I’m confident that all of my mental abilities, both logical and artistic, are owed to the structure of the matter in my brain. “And if it’s all in there, then I see no reason to argue that certain aspects of it will be reproducible on another substrate while others will not. Indeed, for all I know, AC may actually be simpler than AI. Maybe we’ve been creating Artificial Consciousness since 1893 and just haven’t realized it yet because toasters can’t cry.”

On Philosophy, et cetera, Chappell discusses a theory of goals, originally described by Railton, with some similarities to CEV:

Give to an actual individual A unqualified cognitive and imaginative powers, and full factual and nomological information about his physical and psychological constitution, capacities, circumstances, history, and so on. A will have become A+, who has complete and vivid knowledge of himself and his environment, and whose instrumental rationality is in no way defective. We now ask A+ to tell us not what he currently wants, but what he would want his non-idealized self A to want – or, more generally, to seek – were he in the actual condition and circumstances of A. (pp.173-4, bold added.)

The last bit seems to me like a hack to deal with observer-centric goals; if A+ has been constructed, then A+ is not in the actual condition and circumstances of A. This hack can lead to some suboptimal outcomes.

Suppose, for instance, that Fred wants a diamond. Fred has some other concerns too: he would also like to have an emerald, and he would prefer for his personality to remain intact, but mostly he just wants a diamond. Fred can choose between two boxes: box 1 contains a diamond, and box 2 contains an emerald. Fred erroneously believes that box 1 contains an emerald, and box 2 contains a diamond. If we extrapolate Fred+, then Fred+ would want Fred to want an emerald. Then Fred will choose box 1, and receive a diamond, thus fulfilling his original goal. Fred’s lesser preference of keeping his personality intact is not fulfilled.

A better solution would be to tell Fred that box 1 contains a diamond. I am not saying that this is the optimal solution, of course.