The classic thought experiment is the paper clip optimizer. Quoting Nick Bostrom:

> Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans. https://www.huffpost.com/entry/artificial-intelligence-oxfor...

If you substitute "shareholder value", this is basically how corporations already make decisions with effects on human life.

Sure, I once write that companies are artificial intelligence and it still feels true: https://beza1e1.tuxen.de/companies_are_ai.html

Cstross certainly did say that in 2017.

> I'm talking about the very old, very slow AIs we call corporations, of course.

https://www.antipope.org/charlie/blog-static/2018/01/dude-yo...

The fossil fuel industry managed to invent this unhappy process long before machine learning was a thing.

Ender's game model, presumably: the AI helpfully assists a human to build a nuclear bomb / pandemic virus / autonomous killer robot swarm in their basement. But again, I would like people to be clearer about how the threat is supposed to work rather than just making SF references.

Yet we have wiped thousands of species completely by accident, breed some for slaughter and consumption and trap some for entertainment.

> We have not wiped out apes, ants, and most other species.

Of course not all of them, just the ones that inconvenienced us in any way in pursuit of our goals.

It's not exactly trained on a neutral set of data. Engagement algorithms have ensured that a good chunk of content on the web these days is inflammatory and skewed towards the extreme (in fact in the last few months a good portion of the world has actually deemed it illegal for kids to consume because this content is so damaging to a developing brain).

Well, hopefully we end up like them then, and not https://en.wikipedia.org/wiki/Category:Species_made_extinct_... or worse, https://en.wikipedia.org/wiki/Category:Species_made_extinct_...