There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this.
In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
> I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering.
If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.
You'd be surprised how many customers are lead to believe this by a combination of opaque marketing, aggressive sales, and willful ignorance.
I worked at a 2000 person shop where the CTO moved us from 1 on-prem dc (we were begging to add a secondary site for years due to outage risk) to 1 AWS region.
It's not exactly straightforward to run multi-region across an alphabet soup of AWS services without decent configuration / application awareness, paying at least double.
S3 is advertised as having 11 9's. That means in the entire history of S3 they've only lost a handful of objects, and will continue to lose objects at this same rate.
You are confusing the concepts, and applying the 11 nines to the wrong problem.
Durability is about: "If I successfully store an object in S3, how unlikely is S3 to permanently lose that object because of storage failures?"
It does not answer: "Will I be able to access that object, after a rain of Shahab-1 or Shahab-2 burn all data centers in the 3 availability regions across which my S3 bucket is spread out..."
Eleven nines covers ordinary infrastructure failures, not acts of war that physically destroy the underlying facilities.
It's similar to any engineered artifact, like a skyscraper. They have a structural safety factor that defines their resilience to failures that the system is designed to tolerate, it doesn't make them immune to missile strikes.
This is a normal failure mode in American military facilities in the Middle East. Whoever designed an American military facility in the middle east and did not intend for it to withstand missile attacks was incompetent.
AWS has a history of making hand wavy explanations on the robustness of its infrastructure that were misleading at best. See comments about how regions are truly independent only for everyone to find out if us-east-1 goes down you could still be down even if you built in other regions. Those dependencies were not well documented and previously hand-waved away by AWS when it boasted about how its regions were truly independent.
Similar here, there’s a lot of detail that got hand-waved away by a sloppy “yeah we good” puff PR answer.
There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this.
In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.