So what would apes like ourselves do in this circumstance?
Well, we'd be free to play Frisbee and give each other massages.
Add some LSD and some questionable wardrobe choices, and the whole world could be like Burning Man.
Now, that might sound pretty good, but ask yourself what would happen under our current economic and political order?
It seems likely that we would witness a level of wealth inequality and unemployment that we have never seen before.
Absent a willingness to immediately put this new wealth to the service of all humanity,
a few trillionaires could grace the covers of our business magazines while the rest of the world would be free to starve.
And what would the Russians or the Chinese do if they heard that some company in Silicon Valley was about to deploy a superintelligent AI?
This machine would be capable of waging war, whether terrestrial or cyber, with unprecedented power.
This is a winner-take-all scenario. To be six months ahead of the competition here is to be 500,000 years ahead, at a minimum.
So it seems that even mere rumors of this kind of breakthrough could cause our species to go berserk.
Now, one of the most frightening things, in my view, at this moment, are the kinds of things that AI researchers say when they want to be reassuring.
And the most common reason we're told not to worry is time.
This is all a long way off, don't you know.
This is probably 50 or 100 years away.
One researcher has said, "Worrying about AI safety is like worrying about overpopulation on Mars."
This is the Silicon Valley version of "don't worry your pretty little head about it."