I think that is what they are actually looking for. Imagine: You have capital, you invest in a business that is run entirely by computers. You want to scale? Just provision more agents / cores. It's completely autonomous. It's a money printing machine that extracts revenue from customers, with no work required by the owner.
Frankly, the finance world has already achieved this -- they just lend money to make money, with no work involved. Regular businesses are jealous so now they feel they can use computers to do the same thing but in non-finance areas.
Your lack of creativity is not a reason to believe that a super AI is harmless or less destructive than a nuclear weapon. Damage need not be limited to destruction. Introducing doubt is sufficient. Right now you have faith that digital Financial transactions can be trusted. You have faith that computer encryption can be trusted. You have faith that digital certificates will protect you. If an AI can introduce doubt into any one of those systems, that will be sufficient to bring about the destruction of those systems. Imagine a world in which you can no longer use a credit card or Apple pay. Where no digital cash transaction can be trusted or validated. What effects do you think that would have on commerce? How quickly do you think we can return to some trustable means of commerce? Do you think it will happen before your groceries run out in your apartment? Before your grocery store can settle its debts? Before your Amazon ec2 instance runs out of credits?
I still remember when Interstate exits in the US were numbered sequentially. It seems like only 20 years ago they switched to mile markers for exit numbers.
reply