Comments on: The Power of Intelligence http://singinst.org/blog/2007/07/10/the-power-of-intelligence/ The Singularity Institute exists to confront the challenge of powerful AI, both the opportunity and the risk. Thu, 15 Sep 2011 10:21:31 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Sam http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-190041 Sam Tue, 09 Jun 2009 15:22:32 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-190041 Umm...scarcity is an inherent property of the universe. As long as there are sentient beings there will be transactions; as long as there are transactions there will be money. It doesn't matter how much *any* technology improves the world--no technology is going to be omnipotent; no technology can create resources out of wishes. Money is necessary to the survival of intelligent life. Umm…scarcity is an inherent property of the universe. As long as there are sentient beings there will be transactions; as long as there are transactions there will be money. It doesn’t matter how much *any* technology improves the world–no technology is going to be omnipotent; no technology can create resources out of wishes. Money is necessary to the survival of intelligent life.

]]> By: Day of the Squishy Things | Isotropic http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-54355 Day of the Squishy Things | Isotropic Sun, 01 Jun 2008 22:29:40 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-54355 [...] A Premature Word on AI (at Overcoming Bias), Eliezer Yudkowsky referred to an earlier post of his, The Power of Intelligence (at The Singularity Institute), which starts out with: In our skulls we carry around 3 pounds of [...] [...] A Premature Word on AI (at Overcoming Bias), Eliezer Yudkowsky referred to an earlier post of his, The Power of Intelligence (at The Singularity Institute), which starts out with: In our skulls we carry around 3 pounds of [...]

]]> By: Richard Hollerith http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-15425 Richard Hollerith Thu, 22 Nov 2007 03:58:21 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-15425 <i>A superior machine can invent all kinds of workarounds to avoid the inconvenience of our flimsy, built-in “ethics”</i> But a correctly built superior machine has no goals or preferences other than what the designers intentially give it, so it will not want to invent a workaround. I hope you will read more of Eliezer's writings on AI. For example, <a href="http://sl4.org/wiki/KnowabilityOfFAI" rel="nofollow"> Knowability of AI</a> A superior machine can invent all kinds of workarounds to avoid the inconvenience of our flimsy, built-in “ethics”

But a correctly built superior machine has no goals or preferences other than what the designers intentially give it, so it will not want to invent a workaround.

I hope you will read more of Eliezer’s writings on AI. For example, Knowability of AI

]]> By: denis bider http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-15129 denis bider Tue, 20 Nov 2007 21:54:20 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-15129 Ugh. "Ethics". What we're talking about here is no ethics. We don't want the supermachine to be "ethical". We ourselves are not "ethical", especially not towards inferior things. No one (except in limited cases) expects us to be ethical towards inferior things, and no one (except in limited cases) actually enforces our being ethical towards inferior things. Our "ethicism" in this regard is fraught with inconsistency. On the one hand we have PETA and the pet police. On the other hand we slaughter pigs and cows and create chicken who can't stand upright because we've made their breasts grow too big. We go to the supermarket and buy a bag of shrimp that have been slaughtered by the billion. We have no bad thoughts about that. We operate on no "ethics". What you're talking about here with "friendly AI" and "ethical AI" doesn't involve any lofty principles either. The basic principle you wish to imbue your AI with is something like: "thou shalt not harm thy creator". This is pure human selfishness, and our ability to imbue a vastly superior being with that concept may be very limited. After all, nature has imbued us with a desire to have sex in order to reproduce, and so we invented condoms. A superior machine can invent all kinds of workarounds to avoid the inconvenience of our flimsy, built-in "ethics". If someone succeeds in building strong AI, and the creature doesn't kill itself immediately upon coming into existence, then it's pretty much over for the human race. So far we've argued against religious believers that we can't see god. Well, if strong AI does come into existence, there won't be any doubt, we'll all see one. That said, I don't really see any incentive for a VC to fund something like this. A VC is a creature that operates and makes sense within the confines of a certain environment. A VC, as an organization, doesn't necessarily have any incentive to change or revolutionize that environment. Nor does it have the authority to do it. Your best bet for funding would be rich people who can dispense with their own money without having to answer to anyone. Ugh.

“Ethics”. What we’re talking about here is no ethics. We don’t want the supermachine to be “ethical”. We ourselves are not “ethical”, especially not towards inferior things. No one (except in limited cases) expects us to be ethical towards inferior things, and no one (except in limited cases) actually enforces our being ethical towards inferior things. Our “ethicism” in this regard is fraught with inconsistency. On the one hand we have PETA and the pet police. On the other hand we slaughter pigs and cows and create chicken who can’t stand upright because we’ve made their breasts grow too big. We go to the supermarket and buy a bag of shrimp that have been slaughtered by the billion. We have no bad thoughts about that.

We operate on no “ethics”.

What you’re talking about here with “friendly AI” and “ethical AI” doesn’t involve any lofty principles either. The basic principle you wish to imbue your AI with is something like: “thou shalt not harm thy creator”. This is pure human selfishness, and our ability to imbue a vastly superior being with that concept may be very limited. After all, nature has imbued us with a desire to have sex in order to reproduce, and so we invented condoms. A superior machine can invent all kinds of workarounds to avoid the inconvenience of our flimsy, built-in “ethics”.

If someone succeeds in building strong AI, and the creature doesn’t kill itself immediately upon coming into existence, then it’s pretty much over for the human race. So far we’ve argued against religious believers that we can’t see god. Well, if strong AI does come into existence, there won’t be any doubt, we’ll all see one.

That said, I don’t really see any incentive for a VC to fund something like this. A VC is a creature that operates and makes sense within the confines of a certain environment. A VC, as an organization, doesn’t necessarily have any incentive to change or revolutionize that environment. Nor does it have the authority to do it.

Your best bet for funding would be rich people who can dispense with their own money without having to answer to anyone.

]]> By: Steven Thomas http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4592 Steven Thomas Sun, 09 Sep 2007 08:04:42 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4592 This AI is going to have to be very smart to overcome the problem of limited resources, and smarter still to overcome the conflicts that arise from the other functions of our brains, such as instincts and passions. Our planet can only accommodate so many people. Will this AI teach us how to live simply and fairly? Will this AI solve jealousy, gluttony, xenophobia, religious fundamentalism, recklessness, vengeance, and other basic tensions of primate social structures? Can it solve these problems, and leave us human? This AI is going to have to be very smart to overcome the problem of limited resources, and smarter still to overcome the conflicts that arise from the other functions of our brains, such as instincts and passions. Our planet can only accommodate so many people. Will this AI teach us how to live simply and fairly? Will this AI solve jealousy, gluttony, xenophobia, religious fundamentalism, recklessness, vengeance, and other basic tensions of primate social structures? Can it solve these problems, and leave us human?

]]> By: Davy http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4585 Davy Sun, 09 Sep 2007 06:03:39 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4585 Perhaps we should first create an ethical foundation that guides our intelligence. Perhaps we should first create an ethical foundation that guides our intelligence.

]]> By: Grant Czerepak http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4567 Grant Czerepak Sun, 09 Sep 2007 02:55:23 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4567 The concept of migrating intelligence from the strata of carbon based organisms to another strata is still hard to conceive. But it is obvious that even though the body or id may change, the soul and spirit or ego and superego, depending on you perspective, are still inseparable at this point. What I am trying to say is we may create an intelligence, but we also have to create an ethical foundation that guides that intelligence before we unleash it. And the more I learn about the relativistic reasoning of the human mind when making ethical judgments the less faith I have in our ability to create what we would deem an ethical AI from our own perspective. Half the world could consider the AI friendly and the other half could consider it an abomination. The concept of migrating intelligence from the strata of carbon based organisms to another strata is still hard to conceive. But it is obvious that even though the body or id may change, the soul and spirit or ego and superego, depending on you perspective, are still inseparable at this point.

What I am trying to say is we may create an intelligence, but we also have to create an ethical foundation that guides that intelligence before we unleash it. And the more I learn about the relativistic reasoning of the human mind when making ethical judgments the less faith I have in our ability to create what we would deem an ethical AI from our own perspective. Half the world could consider the AI friendly and the other half could consider it an abomination.

]]> By: Life, the Universe, and Everything » If Singularity Were Like Nanotechnology http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-1323 Life, the Universe, and Everything » If Singularity Were Like Nanotechnology Thu, 26 Jul 2007 20:22:11 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-1323 [...] take into account the need for a reliable FAI brand name… wait, this has apparently already happened [...] [...] take into account the need for a reliable FAI brand name… wait, this has apparently already happened [...]

]]> By: Jeffrey Herrlich http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-893 Jeffrey Herrlich Thu, 19 Jul 2007 13:54:32 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-893 No they're not stupid, but it appears that the majority of them *don't* truly understand, or at least they don't accept the situation viscerally. It's definitely an unfortunate situation at this time, hopefully it can be changed. No they’re not stupid, but it appears that the majority of them *don’t* truly understand, or at least they don’t accept the situation viscerally. It’s definitely an unfortunate situation at this time, hopefully it can be changed.

]]> By: Fredrik http://singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-885 Fredrik Thu, 19 Jul 2007 07:02:30 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-885 Eliezer: What a FANTASTIC story! Thank you! Eliezer: What a FANTASTIC story! Thank you!

]]>