This strikes me as the only place where progress is being made in this whole discussion.

In what order are the various extrapolations done?

I’m perfectly happy with the “knew more” extrapolation, it seems reasonably well-defined, and it’s what I would want, personally, for me and everybody. I even like that it happens first, before, as you put it, “considerably more complicated and recursive processes”.

But does it happen first?

According to the poetry, “knew more” etc. is to be “interpreted as we wish that interpreted, extrapolated as we wish that extrapolated.” Doesn’t the “interpretation as we wish that interpreted” constitute a RATHER complicated and recursive process, to say the least?

It would be really nice if you could break down the order of operations more explicitly.

Other people, of course. Tricking someone by creating a logical-sounding reason for something that is actually bad has tons of uses.

Who would play such nasty tricks on people as to create rationalizations? How can we weed them out? Do their leaves look different from the good plants?

Human thought appears to be nothing but a vast and expanding field of attractors, just like the evolutionary fitness landscape.

Bayes save us!

Ditto.

Tom: “I highly doubt that CEV will take rationalizations into account”

Maybe, maybe not. I don’t have enough information to answer this. I think it’s a risk though.

I highly doubt that CEV will take rationalizations into account, as rationalizations are created specifically to fool people, and I don’t think anyone would want to be fooled in the abstract into accepting undesirable consequences when they’re totally unnecessary.

Dangerous attractors lurking out in extrapolated volition space.

I suspect that this is a universal reaction in all evolved cultures where eliminating death has been raised as a possibility, but not yet implemented. Evolution will tend to build in old age as most creatures will die before reaching it anyway, and generations upon generations of the evolved intelligences will try and reduce the pain of death by concocting these kinds of rationalizations.

If it was that exact response, I’d be pretty damned surprised! My blink reaction would be that the CEV wasn’t extrapolating far enough because it had just produced a characteristically 20th-century response. I would also of course have to question the whole CEV paradigm and whether it was doing anything like what I thought it did. But I might ask a trustworthy friend to peek at the justification to see if there was some blindingly nonobvious reason involved.

When I read the part of your case 2 which said “sound waves caused some processing to take place in your brain which caused you to change course”, your unusual wording led me to assume you were referring to some unusual mind-control sound waves. But if you were only referring to the ordinary sound waves of speech, then your case 2 is really an example of persuasion instead of force. I still maintain that there’s an obvious and important distinction between persuasion and force.

“Would you also consider it thought control if I said, “stop, you’re about to burn your hand”? ”

That would be a clear example of persuasion, not force.

“You say you want the MCAI to develop a formula based on examples we give it. Who is “we”? The AI programmers? The philosophy department at your university? The United Nations? Who do you think is qualified to write the laws that dictate what RPOPs can and can’t do for the rest of time?”

Who would be qualified to hardwire CEV into the first RPOP? Who has the right to develop powerful AI at all? In my opinion, whoever develops a powerful technology has the obligation to help avoid problems it may create.