Seems like a bad design though to put both on the same network. Voice is relatively low bandwidth but more important. Also voice is typically more localized, a controller in SFO isn't talking to an airplane in NYC.
Do we know that voice and radar were lost though? How did the flights in the air land? I thought the thing that broke was the handover between different regions/spaces/controllers?
The ATC operating at airports, like ground, tower, departure, etc - are all local and simply transmit voice over AM frequencies. They were not affected so flights could land.
Regional/sector ATC (managing flights en route that left the departing airport airspace and have not yet arrived to the arrival airport airspace) are also transmitting in AM but the actual operations are not where the transmitters are (as a lot of them are in the middle of nowhere). The voice/data transmits back and forth from the actual op centers to the transmitters/receivers over the network. Which is what went down.
Actually some short flights (like flight schools or private pilots on short flights) were allowed to continue to operate, as they never left the airport or arrived at an airport so close that they never needed to talk to ATC managing a sector.
>Actually some short flights (like flight schools or private pilots on short flights) were allowed to continue to operate
Are you sure about that? A pilot buddy of mine said this covered even things that didn't rely on radio communications like flights originating and arriving at small rural airfields, vintage planes without radios, gliders and hot air balloons.
The radars are actually pretty low bandwidth. The big center radars only turn 5 times a minute. Say you have a few hundred targets in the radar volume, each one gets a position, altitude, transponder code, mode S data, etc, maybe 64 bytes tops. You’re talking maybe dozens of kilobits per second.
This reads very human but "that only demonstrates plausibility, not truth." reads AI ;) Are you imitating ChatGPT? :-) But I agree. This could be a routing issue or some other interaction. Could also just be overload of traffic between these networks of different capabilities. At you say, probably not rate limited in the interface between the systems.
It's not hard to screw up networks. I remember our IT person introducing a loop into a simple office network that took like two hours to troubleshoot.
You'd think they would have a) noticed sooner b) made the connection between the last change and the issues. But easy to be a backseat quarterback as they say.
I'm a human that is still salty that AI has stolen the em dash from me, along with a whole lot of rhetorical seasoning that used to make text more interesting, and now just makes it as slop. Grr!
Plausible I guess. Essentially a denial-of-service attack from a new device connected to the network. The article doesn't give more details but a device in the network can ... send traffic into the network ... and a device connected to a high speed link in a network with many old legacy/slow devices ... could cause problems.
The router was apparently a Juniper router. L3Harris I guess buys them and install/configures/whatnot. They don't make them.
There are methods for building complex systems our of smaller pieces. Humanity has done this forever. An internal combustion engine is very complex. An airplane is very complex. You put a boundary around your system, you build it to a spec, you test it to a spec, you define the behaviour across anything you can see. When two engines fail on the airplane it can maybe still at least glide. Maybe there's an ejection seat. Because someone is doing the system thinking with some level of discipline.
A brick is very complex. A bearing is complex. A bolt and a nut and a washer are complex. An integrated circuit is complex. We build stuff out of that. Like buildings that don't fall when you sneeze on them.
In software we use duct tape - at best. But that's not just Microsoft, that's pretty much everyone. With maybe few exceptions.
Software systems are by nature more complex than physical systems. They also lack any tangible interfaces beyond code and diagrams and collective knowledge in the minds of the humans who build and maintain them.. The rapid evolution of software is no help either..if anything AI is accelerating that rapidity as well.
I have lived through and debugged cascading failures and large scale distributed systems.
The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.
Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).
A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.
We took some of the hardest problems in computer science: distributed systems, scale, concurrency and networking, and made them the standard for every To do app.
This is the inevitable consequence.
When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.
People expect software to be feature rich, complex and solve all the world’s problems. And if you don’t do that, the vibe coded competition will beat your cleanly architected product commercially.
It's not even better though. It doesn't have more features because it's a distributed system. It isn't faster because it's a distributed system. Maybe it has a bit higher uptime but nobody was paying too much attention to that anyway, except when it's down. And it's much better for your PR if, when it's down, it isn't down for everyone at once.
Managers and marketers are the ones asking for all those useless features. And god forbid they run out features to ask for, because then they start asking for architecture v2.0 projects.
I'll be the downvoted comment, since HN doesn't like to hear this, but the other flavors of engineering do stuff we don't because they're regulated.
My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.
I can't think of any other engineering discipline where demand might increase by a factor of a million in a few seconds because some DNS record was propagated incorrectly or something - regulation won't magically solve the issues unique to software. Maybe it's doable with vastly increased costs (a classic would be running fully redundant 1:1 warm failovers, which nobody could economically do at the cloud provider level.)
That's not to excuse incompetence or whatever, just to note that the benefit of software is precisely how easy it is to change, which will always lead to more problems.
Electrical engineering of the power grid has the exact same failure mode: cascading failures of interconnected power grids can cause loads to increase by factor of a million in a few seconds.
I remember a power outage in 2003 that knocked out power for 55 million people and took over a week to restore in many places.
Note that buggy software was a major contributing factor! The software was designed to, and would have, detected a deteriorating system an hour before anything broke, which would have provided ample time to intervene. However, a race condition caused the whole monitoring system to display stale data and then crash, including the redundant backup. This wasn't the only problem by far, but it was one layer of swiss cheese.
Also note the power companies have clear procedures to reboot the grid, known as a "black start[up]". And they test these procedures, though without taking down the grid.
I found out they have prepared, in advance, a list of which disconnects have to be open at subatations in order for power stations to supply each other's startup needs (for stations that are not capable of unpowered startup) while supplying no or minimal actual load. There must be a software analogy to this. Does your company have a plan to bring up the system on a rackful of blank servers and a database backup? I doubt it.
Which is the kind of exceptionalism that really creates these problems. Software is the only industry that allows people untrained in running global infrastructure to run global infrastructure.
Given that people on reddit claimed that in the entirety of New Zealand no real estate transactions can be settled, causing cascading failures [1]...
I think this might actually be the incident pushing governments to designate large online services such as MS 365, Gmail et al as critical infrastructure and force companies to massive, yet-unseen levels of transparency and regulation.
The EU is already itching to act ever since the Trump admin forced US Big Tech to terminate service to the ICC judges, and this here was a square own goal for Microsoft.
The other option is to not outsource everything to the cloud. Yeah, it is more convenient ... but in these times, being a bit more self reliant is not a bad thing.
In these times, paying more for that is not an option. Servers with lots of RAM are 5 times what they cost a couple of years ago. Cloud is a huge bargain today compared to running your own infrastructure.
The problem is that you are only as strong as your weakest link. At some point, something somewhere will be architected poorly and it will one day line up perfectly with 2 other bugs that were incredibly hard to predict and then you're stuck in Microsoft's current situation.
Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.
This is the same as what's been happening with entire industries: Build for the critical path. Eliminate redundancies. Eliminate resiliency. Lean staffing. Just-in-time manufacturing. Single-vendor mandates.
In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.
And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.
So what's your answer? Just ask the giant megacorporations siphoning billions of dollars a year into the pockets of oligarchs to pretty please stop ruining things for everyone else while they enrich themselves?
This attitude of "regulations don't work" "taxes will just cause wealth to flee" "increasing the minimum wage will just be inflated away" is all propaganda. Propaganda meant to make us believe all we can do is give in to the megacorporations and the oligarchs who lead them, and let them do whatever they want.
How do you define "good coder"? I've yet to see LLMs that write better than what I consider the best human code. I think Michael Hashimoto showed that not too far back where getting LLMs to optimize for a problem resulted in code better than naive but much worse than expert and that's before even talking about how that code looks/reads.
I'll admit I'm not on the very cutting edge models but LLMs make weird tradeoffs and lack creativity. They will sometimes write marginally better but way over-complicated code for no good reason. That can have performance implications and they're generally not "aware". They can gain awareness via profiling but again run into limited creativity or big picture understanding. They are good with cliche but not as good with novel situations. With human guidance this is still productive/useful but as the systems grow more complex they are less able to reason correctly about them.
I would say that in certain situations they outperform the average software developer. They definitely outperform in speed and output. So they're like a maybe 1600 ELO chess player for software. I.e. they play chess better than all non-chess players. They play chess better than many chess players. But they're not "master" or "gm" level across the entire discipline. But they can play 100k chess games in the time it takes me to figure out my move...
That makes perfect sense. It's all a question of what things you attuned to. To you they all look the same but to others not. Jews in the US faced a lot of discrimination. Even into the 1970's you'd find things like housing complexes that would not accept Jewish people. Country clubs that would not accept Jewish people. In the 1950's and 1960's universities would have "quotas" for Jews. Even up to the 1990's (and maybe even today) there were clubs and groups that discriminated against Jews. Ethnic Jews are somewhat genetically distinct and so do have somewhat different "looks". It can be minor stuff to you, certainly not as large differences as between people with an "asian" look and "white" (and what you consider "white" is probably different than what some other people consider "white).
High rates can make the currency more attractive. Not less. Tends to strengthen the USD. So the next conclusion is also false.
But yes, higher yields can be related to demand, but they're not simply a function of demand. They relate to things like inflation expectations, and so the short term rates set by the Fed, and the risk premium. In this case likely the higher rates are not high due to demand but rather due to inflation expectations and the Fed having raised the overnight rate.
Pre-1967 Israel didn't control the West Bank or Gaza. So most of the bad stuff people claim about Israel today didn't really exist.
Yet all its neighbours wanted to destroy it.
Lebanon who in theory should have no dispute with Israel bombarded it with ten's of thousands of rockets, mortars, anti-tank munitions etc. After Oct 7th.
The bit you're missing in your overly simplistic analysis is that Israel is really surrounded/attacked by a single religious/political ideology. There is no "everybody else here".
And what you are missing is that Israel acts don't call for friendship even in the people that would want this friendship.
Israel has stolen as much military and industrial secrets from US as they could grab. Has violated every international law. There are strong suspicions to be behind Epstein honey trap for riches and politicians all around USA and Europe. Has funded Hamas. Has an horrible relationship with human rights, longer than the bible. Has called non Jews "human animals", and acted accordingly, even to the point of doing competitions about "who can destroy more kneecaps or shoot more testicles in a week", among starving people that were trying to obtain food none less. Has treated 5Yo children like partridges in a hunting party. Has killed thousands of journalists, doctors, nurses and NGO staff without even trying to pretend that they care. Has put spy devices in friend's phones each time they could. Was caught messing with the elections in UK. Has used, discarded, and left each politician that supported them, alone and hanging dry in the elections, from Biden, to Trump, to Merz.
US does not want the demise of Israel, but Israel does not even try to pretend that they respect the opinion of US. Even after all this time, is all grab and cash. And US is always cleaning the mess that they left behind.
And the only thing that we can hear from them is "Why everybody hates us?" Really?. Clean your acts first.
China has stolen x1000000 secrets from the US. Everyone spies on everyone. Israel relatively less. So that assumption is just baseless.
"Has violated every international law". Baseless. Violates less "international laws" than most western countries, not to speak of the rest of the world. I mean "international law" isn't really a thing, it's a series of treaties.
Called non-Jews "human animals". Just the Hamas and the perpetrators of Oct 7th. And rightly so. I mean animal is a compliment to those people. (all humans are animals really).
I don't know where you're getting all of this from. But it's an alternate reality. It is not factual. It is putting Israel under a microscope when most of the world behaves much worse. There are dozens of countries with much worse human rights violations than Israel is even accused of. And many of those accusations are outright lies. That's not to say every Israeli soldier is perfect or that Israel is perfect, but it's certainly better than e.g. anything the west did when it went after ISIS/ISIL or Al Qaeda or pretty much any other war waged by the west. And the west is better than the rest of the world by far.
You are just echoing propaganda here. And that's the reality of why people hate Israel. Because they are bombarded by propaganda that tells them to hate Israel. If you actually are able to zoom out of that you will see it as well. That plus antisemitism.
EDIT: You are right that Israel isn't spending a ton of cycles trying to get e.g. Europe and Americans to "like it". Between the choice of e.g. winning the war in Gaza and e.g. giving up to Biden's demands (e.g.) Israel has chosen that it must win the war. The American left, media, and universities, e.g. are structurally biased against Israel. They are regurgitating cold war USSR narratives. This is the result of more than two decades of anti-Israeli propaganda. Israel sometimes has to cut its losses. It can't out compete the resources of Qatar, Russia, and China e.g. who are out to splinter the US by whatever means possible. It could do better but between "making friends" and a decisive win against Hamas which is crucial for its future security it will pick the decisive win.
Look at the media coverage regarding Israel's continued pressure on Hamas since the ceasefire. Israel keeps eliminating Hamas commanders, it's eliminating people who were part of the Oct 7th raid into Israel, it's eliminating infrastructure used for military purposes, it's eliminating people who have held hostages. What the media is reporting is simply that Israel is "violating the ceasefire" and "killing Palestinians". That narrative is false. If you believe it then you're going to "hate Israel" as you are programmed to do. But Israel like any other country has the right to win against Hamas in this war Hamas started.
EDIT2: Also reality is complex. So we do have things like an extremist right wing government who is turning a blind eye to violence against Palestinians in the West Bank. However that tends to be blown out of proportion. We see western media simply not reporting on Palestinian violence against Israelis. Just like they didn't report Hezbollah's attacks on northern Israel. The evacuation of the population there. And generally anything that puts Israel in a possible light as being the one defending itself. So if you consume simply a one sided picture you're going to end up with a one sided picture view. And the wrong view. You won't understand e.g. why Israel went into Lebanon if you never heard about the scale of attacks on Israel originating from Lebanon. Israel isn't always right, and ofcourse attacks on innocents are never justified, but those are some small pieces of a big picture, and you're only getting those pieces. From Gaza you are basically getting 100% Hamas controlled views which the media repeats as facts (journalists, doctors, etc.). When media talks about Palestinians in the west bank it also immediately whitewashes everyone into civilians. In fact there are armed factions operating in the West Bank and a lot of the overall violence (not the settlers vs. random innocent) is towards dismantling those.
Maybe the downvoters can actually provide some real sort of evidence to support their case. If Israel "steals more secrets from the US than China" as is asserted let's see that proven. If human rights in Israel are worse than lessay China, or Saudi Arabia, or Egypt, or Qatar, lets see something to support that. Well- that something doesn't exist.
Do we know that voice and radar were lost though? How did the flights in the air land? I thought the thing that broke was the handover between different regions/spaces/controllers?
reply