Tuesday, December 7, 2010
I've come to believe, as a direct consequence of reading Less Wrong that we're almost certainly all doomed.
To be fair, I've always thought that we the species were a bit doomed, what with over-population and everything, but I used to think that it was a long way away in the unbelievably distant future, and who knows what might turn up in the meantime? Now I think that we're doomed on a fairly short time-scale. If not us personally, then our children or their children.
Anyway, I thought I'd try to compose a proof, because setting out a logical argument and seeing if other people can knock it down is a good way to lose false beliefs.
So here it is:
1/ It is possible to create an Artificial Intelligence as clever as a human being.
2/ Therefore, one day, someone will create an AI as clever as a human being.
3/ This AI will have goals.
4/ It will realize that the best way to achieve its goals will be to make itself cleverer.
5/ The AI will be able to improve itself to become extremely intelligent.
6/ The AI, being extremely intelligent, will achieve its goals very quickly.
7/ Very many possible goals will result in the destruction of humanity if achieved.
It's a bit sketchy. For instance 3 isn't true. It's just that an AI without goals won't do anything, and so whoever built it will tweak it until it does have. Also I can imagine many goals which won't destroy humanity. It's just that they're all fairly trivial. You don't need an AI to achieve them, and I imagine that the creator of this thing will want it to do non-trivial things. So same again.
Also, of course, there are bound to be goals which don't result in the destruction of humanity. Indeed, if the AI creator picks right, the creation of an AI could be a wonderful thing. But I think that it will be much easier to create an AI than it will be to find non-destructive goals for it (and then program those goals!). And it doesn't have to just not destroy us. It has to prevent any other new-born God destroying us.
Pick holes! I certainly don't want to believe this, but I find myself forced.
This feels like a religious awakening. Last time I looked like getting a religion, as a consequence of listening to too much evensong, friends were kind enough to laugh me out of it, and I have always been grateful.
Go for it people. Use any method you like. Even ridicule is good. I positively want to lose this belief. I will be grateful if you can disabuse me of it.
And I'm not trying to claim that all this is in any way my idea, by the way. It's fairly commonplace, it seems. It's just that I can't for the moment see why it isn't also true.
Posted by John Lawrence Aspden at 4:49 PM