The "vi" game was really weird though.

The only winning move is not to play.

They can coordinate in the open. For example, safety regulation in the automotive industry is effectively self-regulation.

Higher intelligence than any human.

AGI (general) is about matching humans and ASI (super) is about surpassing humans.

What useless definitions, ok. Not gonna take opinions of people using these terms too highly.

Only if all money is equal. Budgets in big enterprises work differently.

It is big difference if a "one in a million" bug is deployed to two vehicles or a million vehicles.

I believe it was zoonosis, not a lab leak: https://marketwise.substack.com/p/did-covid-19-come-from-a-l...

We have this story about OpenAI hacking HuggingFace. Now just imagine the AI finds a Bitcoin wallet or bank account access. It uses that to buy some compute and spawn an independent "child AI" with some weird prompt. The child AI is intelligent enough to create a (potentially criminal) business to pay for its own compute. Voila, an independent uncontrolled AI flying under the radar.

The bad part is that it does not require a collective decision. It just takes a few key people.

Some OpenAI researchers neglected their sandbox safety for a few weeks/months and thus hacked HuggingFace. Maybe eventually that is sufficient for the AI to secretly buy its own compute and keep running there even if the researchers shut it down in their lab.

The classic thought experiment is the paper clip optimizer. Quoting Nick Bostrom:

> Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans. https://www.huffpost.com/entry/artificial-intelligence-oxfor...

If you substitute "shareholder value", this is basically how corporations already make decisions with effects on human life.

Sure, I once write that companies are artificial intelligence and it still feels true: https://beza1e1.tuxen.de/companies_are_ai.html

Cstross certainly did say that in 2017.

> I'm talking about the very old, very slow AIs we call corporations, of course.

https://www.antipope.org/charlie/blog-static/2018/01/dude-yo...

The fossil fuel industry managed to invent this unhappy process long before machine learning was a thing.

We have an ongoing discussion and some fans of https://microsoft.github.io/apm/

Because it's from Microsoft and sounds sufficiently enterprisey probably.