The reality for most companies, up until now at least, is that if they manage email servers themselves they will have more annualized downtime than if Microsoft does it. Of course if this outage lasts long enough (who knows what's going on over there), that will stop being true.
IMHO a bigger concern is that Russia hacked Microsoft in 2022 and China hacked Microsoft in 2023. Even that wasn't enough to scare companies off MS.
Basically, MS customers are locked in, the cost of migrating to anything else is huge, and there's a "no-one gets fired for buying MS" herd mentality, so MS has little to lose no matter how hard their product fails, just like Cloudstrike.
There's also the question of what alternative would really be better, or at least more reliable. My impression is that Google's infrastructure and engineering are genuinely better, but I'm recently ex-Google so take that with a grain of salt.
When we self hosted, we very rarely had an unplanned outage. It really shouldn’t happen all the time. Plus I could work planned outages around the company’s schedule.
That's all predicated on having a team of people with the appropriate knowledge of how to securely self host an email system. Some organisations aren't big enough for that to be worthwhile, others have decided to forego that cost to let the cloud providers pick it up.
Email is not a particularly easy thing to reliably host and secure for a sizable organisation, especially if its not your main focus.
Annualized outage minutes is the wrong metric. If my company's email is down for fifteen minutes, twice a day, at 2am and 3am, that's over a week of downtime. But we might not even care.
If I lose a whole day, we lose all the productivity for that day and delay countless upstream and downstream efforts by a day.
That's true, but it's more true for some companies than others. A lot of big companies cross time zones.
I suppose "unplanned outage minutes" would be a better metric. I still expect most companies to be worse on that metric. Most companies are just bad at running services reliably, especially when it's not their core business.
> Elon regularly gets away with way more morally (and perhaps legally) egregious stuff and yet he has increasingly cemented his position. Partly through indirect leverage from his companies and partly from buying access to power directly.
I'm not an Elon fan at all, but ... while he sure sleeps around, I have not heard he slept with subordinates, or Chinese spies.
Musk has 4 children with a neurolink exec that reported to him. She was also on OpenAIs board until 2023, where she fed him confidential info about the company
. The relationship has been ongoing since 2016, but was kept secret until 2022.
He kept a lot of his relationships secret, including offering to pay right winger Ashely S. Clair over 10 million/yr to sign an NDA that would prevent her from ever stating she had his child, until he got a position of impunity that thar unlimited wealth and political favors buys you.
Let's not forget that if almost anyone else did what Musk pulled with Starlink in Ukraine he would get a 5 am visit from counterintelligence at his house the next day.
The difference is that Musk is an actual oligarch and Huang is "just" a billionaire.
You can argue about what the word "spy" really means, but even the source you quote thinks she's an asset for the Chinese establishment who was working to secure access to Nvidia chips for Chinese users, in contravention of US sanctions. A terrible terrible person for Jensen Huang to (literally) be in bed with.
The details are relevant if you care about the story. If this were flipped, it would make a difference if she was working for the CIA, the state department, or Intel.
> If this were flipped, it would make a difference if she was working for the CIA, the state department, or Intel.
Because those are independent from each other in a way that the corresponding institutions in China are not.
Every large Chinese company is under the direct control of the CCP. Working for a large Chinese company means you are working for the Chinese government in the same way as a state department employee (very differently from someone working for Intel, at least if they haven't individually received a National Security Letter).
If hacking one devops engineer's laptop is enough to move $6B of assets out of Binance, Binance is staggeringly incompetent and their customers need to move their funds out right now.
I think a sensible architecture would involve many pools of wallets with different availability. So the majority of funds can probably be completely offline, with only occasional transfers to/from various offline vaults. The there could be different groups of different levels of online funds that are ideally physically separated (individual teams and employees can only access a subset of the wallets).
> I wonder though how a sensible architecture would look like.
Any withdrawal should only ever be authorized from wallets protected by multisig and every withdrawal should go through several machines, running different implementations of the exchanges' verification part of the codebase, replaying the user's entire deposit/trading/withdrawal history and verifying if the withdrawal the user's asking matches his balance/history.
Devs working on "verification before withdrawal" on one of the stack should be forbidden from ever touching the other stack, and vice-versa.
This would prevent SNAFUs like this one for now at least two devs would need to be independently owned before such a heist could be pulled off.
I'm not saying hacks would be impossible. But the "I still $6 bn in 20 minutes" would be much harder to pull.
Another sensible thing to do would be to not have $6 bn on a warm wallet: I really don't think Binance really that much at all times available on warm wallets?
Nothing is 100% secure: but the "one dev gets its laptop owned, hence $6 bn disappear" is just madness and, as a whole (not speaking about crypto here), we can really do better here.
For a start we can do better than thinking: "Nothing can ever be 100% secure, so there's no way to do better than seeing $6 bn vaporize in thin air when a single dev's laptop gets pwned" (I'm know it's not what you're saying: you're asking what can be done better).
So many are going to think: "But if you do that, it still can be hacked because this and that". Which ain't the point: the point is to make hacks harder to pull off, not to make them impossible to pull off.
Binance literally tweeted this themselves [1].
So either Binance is lying, in which case they cannot be trusted. Or their security practices allow developers unprecedented access to wallets, in which case they also cannot be trusted.
The people I personally know who signed the "pacing the frontier" petition (including me), or who called for it in other ways, are sincere in their concerns that things are moving too fast for safety.
I don't believe at all in the idea that cheaper LLMs will somehow "democratize LLMs". Rich people will always be able to buy far more compute than poor people, and we know how to turn more compute into more capability (bigger models, more reasoning and more agents).
The only way to address wealth and power inequality is (in democracies) to vote for governments that will fix it.
I didn't, and still don't, think Google was unambiguously good or evil. They did dirty stuff with Chrome, they do addiction optimization, but they also provide great free services, fight endless wars against spammers, and provide reliable secure infrastructure to a lot of other people, and lots of other good things. I would never have joined the Chrome team, or worked on addictiveness, but I felt fine about working on optimization projects to make Google more efficient, which is basically what I did for my first few years there.
It's pretty hard to live in the world and be completely disconnected from anything unethical. Jesus doesn't seem to demand that (e.g. he didn't tell centurions to leave the Roman army).
> They did dirty stuff with Chrome, they do addiction optimization, but they also provide great free services, fight endless wars against spammers, and provide reliable secure infrastructure to a lot of other people, and lots of other good things.
As mconnor liked to say, there is no free lunch. :)
> I would never have joined the Chrome team, or worked on addictiveness, but I felt fine about working on optimization projects to make Google more efficient, which is basically what I did for my first few years there.
Ultimately, I don't think you can divide your niche from the corporation's primary goals (especially if you think it was your duty to be a good employee), so I disagree with your rationalization but kudos for at least having a moral compass and trying not to lose sight of it.
> It's pretty hard to live in the world and be completely disconnected from anything unethical.
Sure, but I don't think that should be an excuse for being fairly well connected.
IMHO a bigger concern is that Russia hacked Microsoft in 2022 and China hacked Microsoft in 2023. Even that wasn't enough to scare companies off MS.
Basically, MS customers are locked in, the cost of migrating to anything else is huge, and there's a "no-one gets fired for buying MS" herd mentality, so MS has little to lose no matter how hard their product fails, just like Cloudstrike.
There's also the question of what alternative would really be better, or at least more reliable. My impression is that Google's infrastructure and engineering are genuinely better, but I'm recently ex-Google so take that with a grain of salt.
reply