“Open Letter” on AGI?
September 30th, 2007 –
With the goal of eventually spawning an AGI Roadmap, should we craft an open letter highlighting the potential promise and peril of Artificial General Intelligence?
Thanks to Michael Anissimov for creating the initial draft and also to a number of other SIAI supporters for their contributions, we have a start on an AGI Open Letter.
Please reply below (or email me: bruce@singinst.org) if you think the open letter idea is worthwhile or not and/or if you have ideas for improvement. I plan to tally results and incorporate as many suggestions as possible. Thanks!
My opinion: cut the two paragraphs in the middle beginning “The prospect and implications…” and “Other futurists…”. They don’t really add anything and read like futurists congratulating themselves for their cleverness.
Nice letter though. I’m curious to see if there is enough consensus among thinkers on this topic to develop such a roadmap.
Thanks, Derek… I’ll consider this. Feedback has been that the letter may be a tad long for its purpose — call to action for an AGI Roadmap.
when this first came out I recall there was some comments along the lines of “people are already convinced one way or the other about the possibility of AGI so it’s not very useful”. I think that that this letter with the backing of scientists would convince some people or at least help support arguments for AGI development. Getting signatures is not going to waste loads of researcher time so I’d say it’s worth it.
If this letter is intended for the general public, I think it will fail, because it uses too much technical language.
It should be rewritten to be understandable by someone who does not know what AI is. If you ask your mother or your local politician, they will think that AI means Amnesty International.
The text is trying to explain not only what the singularity is, but also some of the potential consequences this has, and why this is important to society as such.
In an attempt to do that, it uses as many terms as possible, which are all completely new to “normal” people:
- AI
- narrow-purpose AI
- AGI
- self-improvement
- narrow-AI
- “Human-Level AI”
- Singularity
- futurists
- singularities
- mathematical physics
- event horizon
- outside observation
- Intellectual Revolution
You can at best hope to introduce one or two terms, but asking the audience to learn and understand so many new terms and concepts is simply unrealistic.
Imagine that you are trying to explain this to a good friend, who thinks that AI means Amnesty International.
With this frame of mind, you’ll quickly realise that you need to drop all the technical terminology, and just try to explain these things using concepts that are already familiar to such a person, and introduce as few new terms as humanly possible.
Otherwise, the other person will just think you are crazy to expect them to understand all that information – remember, it took you *years* to learn and understand these things.
Bruce,
I think that Asger has a good point. Maybe a good strategy would be to design a small series of open letters. Make one that is targeted to AI experts specifically – it could leave out a lot of the introductory material, and include some more advanced content. In the real world, an open letter full of PhD or Masters signatures is going to to carry more weight by comparison. However, I think that it could only help to have as many supporting signatures as possible. So perhaps an additional open-letter could be drafted – one that caters to the layman in particular. It could include the basic introductory content, and be open to signatures by anyone. Then, the two (or more) open letters could be presented together. I think that this strategy would produce the best results. In any case, I definitely think that it’s a project worth doing. Scientist’s Open Letters can have a lot of impact. Look at the nuclear weapons one.
SIAI could also draft a single (or a set) open letter regarding the technical feasibility of a reliably beneficial AGI (a friendli-ish AI). To be presented alongside the others. It might be slightly harder to garner signatures though, as there seems to be a systemic bias that leads many smart people to think that a Friendly AI is impossible. (I don’t get it, but there it is). But it would probably be worth a try. BTW, a Friendly AI doesn’t have to provable in order to turn out Friendly. Godel showed that some *true* statements cannot be proven. So, even by default, it could be *true* that a Friendly AI design will always remain Friendly, even though it can’t be proven mathematically. And that’s starting from the dang baseline; and we don’t have to start from the baseline. Even if Friendliness isn’t 100% mathematically provable, we can steer the probable outcome arbitrarily close to 100% – especially with the advent of a self-strengthening goal-system. A 99.9999999999999999999999% probability of sustained Friendliness would suffice for me. And Friendliness may turn out to be mathematically provable anyway, we just don’t know right now.
Interesting… thanks for the multiple letter idea. I’ll think more about this. Of the two types of letters, perhaps this more technical letter would be best to focus on first?
“Of the two types of letters, perhaps this more technical letter would be best to focus on first?”
That’d probably be what I’d recommend. Seeing a list of supporting PhDs might further encourage laymen and newcomers to take the issue more seriously and dig a little deeper.