Honestly I was surprised Apple stayed up during the Microsoft outage as it affected so many interconnected services and infrastructure. I presume that the Cloud providers are shifting some of the workload around and caused other critical services of their customers to fail.
I was AT the Apple store buying a replacement airpod while the exchange outage was happening.
Apple was fine, but this poor old woman was at the genius bar trying to get some accounts connected. The genius at the Apple store was trying to explain the outage - and it just was NOT clicking for the woman. He must have said 4 variations of "Yes your accounts SHOULD connect, but Microsoft is having a major outage right now so they do not."
Apple is notoriously good at avoiding these widespread outages, by doing things independently regardless of industry trends. I am pretty sure they don't outsource their hosting to Microsoft.
Even if they had been heavy Azure users, the impact to Azure was basically 0. Only Exchange Online and connected services were down. Even OneDrive, SharePoint, and Teams worked in all the parts where they didn't interval with Exchange (e.g. Teams Calendar could show your pre-scheduled meetings and allow you to join them, just not create new meetings).
Isn’t Patch Tuesday a simpler explanation? Something deep needs to be patched and rebooted. Or a coordinated malware removal effort, where anything less than full reboots cause propagation.
Do you really think that when these events happen, every single computer is patched at the SAME time ?
Does that sound intelligent ?
Does this happen in other industries ? Does the electric company shutdown every single transformer/station/big machine i dont know the name and perform servicing at the SAME time ?
Amazon's internal direction on how they scale out AWS and build new features is heavily guided by "what will it do to one of our biggest customers that has a secret code name but everyone knows its Apple".
That's not the entirely of their cloud exposure, however. A ton of their stuff runs in AWS, I wouldn't be surprised if they had things on other cloud providers as well.
The Microsoft outage wasn't an Azure outage anyway, it was specifically tied to Exchange Online. No one running their own stuff in Azure had any issues during that event
Mostly true, but not entirely. Azure Communication Services (ACS) was down alongside Exchange Online and the other Microsoft 365 applications. I verified this by testing upwards of a dozen times throughout the outage window. There was absolutely no indication made by Microsoft though that ACS was impacted in any way.
Production servers should be hardened, immutable, and remotely attestable appliances. Everyone should be able to prove every line of code in memory at all times, with access to secrets gated on signed quorum approval from multiple reviewers. They should not have dev tools, ssh, a shell, and certainly not capable of blindly modifying themselves at runtime if a single admin of a Puppet control server asks them to.
Such tactics mean a single compromised engineer laptop means a compromise of production, and thus a data breach is all but inevitable.
I would disqualify anyone seriously suggesting Salt, Puppet, Ansible or otherwise advocating for live-mutating production servers in a sysadmin interview today.
I understand and subscribe to some of the things you say but:
> Such tactics mean a single compromised engineer laptop means a compromise of production, and thus a data breach is all but inevitable.
I hope you don't believe that an immutable image cannot be compromised. The commits done to the build system come from the same engineer laptops and, in a world where CVE are published ao frequently, said compromised image could be deployed largely in only a few minutes hours.
Also there is something to be said with lack of debugging tools. While I agree that reducing the surface of attack is the way to go, I am pretty sure the reason it takes so long for some of the big infra outages we have experienced from various cloud vendors comes from the inability of engineers to log on some systems, observe and potentially fix temporarily by hand to stop a race condition. There is often a race condition and serpent eating its own tail situation involved and logs might not even reach the monitoring infra anymore.
Going back to manage pets is not the way to go but I am not entirely certain removing ability for a subset of engineers to log on systems is exactly the way to go.
> I hope you don't believe that an immutable image cannot be compromised.
Obviously, but I will trust a unikernel with just kernel/init and an unprivileged sandboxed daemon to have dramatically less attack surface than a redhat system. Reducing attack surface is the job.
> The commits done to the build system come from the same engineer laptops and, in a world where CVE are published ao frequently, said compromised image could be deployed largely in only a few minutes hours.
That is what signed commits and signed reviews are for, and a CI that mandates both. I mentioned quorum control and this is the method I always use. I founded the only Linux distribution that does this.
> Also there is something to be said with lack of debugging tools.
In some cases one may wish to expose a custom, unprivileged, and sandboxed ssh daemon that provides basic logging and read only diagnostic tools to admins. This is usually fine. Hardening and observability can co exist. No need for a real ssh and huge traditional desktop linux shell interface attack surface.
But if I want to do major debugging with dangerous tools in prod, which should almost never be required, it should have to be with consent of another team member to add a -new- image to the fleet with those debug tools, temporarily.
This would only be required when the only way to debug is with production secrets, but the default should always be to reproduce on a duplicate system with dummy secrets.
Talos Linux, Stagex, and my own operating systems I have custom made for various past employers.
Custom operating systems are not scary. Make a UKI so your entire OS is just a kernel, init, and a target daemon payload. Treat servers like firmware. The less code in memory the better.
That said, I am working on an general purpose hardened server OS to be released soon publicly covered in my last talk https://lance.dev/talks/bsm2026
Interesting. This is the first I've heard of broad service interruption at Apple in a while, as I can't even remember the last time. Do you suspect something about puppet or their overall security posture to be implicated in this?
Hell they were more stable than all the other clouds so far. Maybe a giant cluster of redhat computers managed like 2007 is actually pretty robust compared to super kube control plane v42 with 100 add-ons and layers thrown in on top.
They just throw a massive number of engineers and money at the existing tech debt while creating more of it. Which -can- keep up with maintenance and a lot of big companies with money to set on fire do it this way. At least until the AI shock is making engineers have to fix all that tech debt at once. Then complexity and mutability are a massive liability no humans can deal with fast enough.
Blackhats and AI only care about the weakest links. iCloud is where all user data lives, and still managed using prehistoric tactics that I strongly suspect will not survive AI assisted blackhats targeting systems engineers for very long.
Apple is making the same stupid mistakes Binance did. Binance was just more profitable to hack, for now.
You mean they should've made disaster recovery plans ahead of time? That requires experience and a certain level of know-how, in a company that has just decided to "voluntarily retire” a quarter of its most experienced workforce. There was a lot of things they could've, should've, and would've done.
I came up with another theory: maybe it was internal load, server to server communication. For example email traffic between Outlook mailboxes (or some kind of API calls). Once the queues were filled up, every server added back to the pool just increased internal load by hammering other servers.
Restricting external access wouldn't solve this issue.
I guess adding a config option for throttling queue processing is something any distributed system should have.
It’s like George Carlin said, all these people attend to the same social environment, they know what’s good for them, they don’t need to meet in a hideout to fix prices to collude.
I don’t understand why someone would design a pool system that can harm swimmers with chlorine and the only safety is an automated email notification, and then blame the email provider and not their own system design for it.
Your work deducts a fixed amount of tax from your paycheck, but every year you’re expected to fill out a series of forms showing what kind of default tax deductions you qualify for and everything from stocks to retirement savings (which is done privately by individuals) have additional tax rules, when you add the multiple levels of taxing (federal, state, city, and school at least) everything gets complicated really quickly.
But if you get it wrong they come after you. So we know they check the results. It would be a lot easier for them to send you what they think you should pay and then each individual gets to decide if they go with that or file a manual return to justify a different amount.
At some point these GPUs will get ITARed and it’s going to be clear how critical of an asset those are. Some companies don’t understand how export controls work and how much control the government can exert and how bad it can get if someone tries to walk around it.
Maybe they mean inbound international flights may be allowed to take off from their origin, on a case by case basis. Usually a clearance is required to take off from both the origin and destination.
Unfortunately, the people abusing these are very persistent and adding more checks will just increase the sophistication of such attacks which works against the interests of the target.
This stops disguising yourself as a legitimate service.
reply