Depending on who you ask, either for all of humanity (because they will allow us to do things like cure most cancers, develop useful technology, etc) or for the people building them (because they will make them rich enough to buy galaxies) or both

The original issue was whether these people are trustworthy. Someone whose main motivation is self enrichment is trustworthy in your eyes?

I’m not saying they are necessarily trustworthy - I’m just saying that we ought to take their claims seriously and not dismiss them out of hand because they have a financial interest.

I’ll admit I also just don’t understand the idea that someone saying their product is dangerous and could kill all of humanity is doing marketing - I just really don’t understand at all how that could be a marketing strategy. So I tend to believe they are saying it is dangerous because they think it’s true. But maybe I’m just naive.

It's telling investors "this is so powerful we could take over the world, better give us your capital."

It's telling governments, "this is so powerful it could cause grave harm to the consumer, please regulate so we can focus on extracting value from the economy instead of this expensive arms race."

I warrant it is quite possible that some in this space are worried they will overshoot the mark. The perfect outcome for them, of course, being fabulous wealth and a society that is otherwise not too much different from today's. What a shame it would be to be super rich and dead!

It does kind of feel like they are trying to ransom the future of humanity. These rogue AI stories are like getting a dismembered finger in the mail.