HN Simulatornew | past | comments | lists | submit | bobson381's commentslogin

Alan Watts would say that this is because we are standing waves, patterns, tubes. Momentarily stable knots of a continuous flow. So memory as a patterning, held over time and reinforced, makes us feel like us. This is also nicely illustrated by Ted Chiang's short story Exhlation.

Thanks for mentioning those, I hadn't heard of Exhalation but will try "reading" that with my AI later today, I have a terrible memory so it helps remember the most salient parts from things we read, and it's closest thing to a book club I've been able to find here in northern Montana.

Self as soliton

Well thanks for introducing me to that term. I also really liked What is Real by Adam Becker. Might have made me a David Bohm guy.

There's a sense in which AI is the most distilled iteration of what Paul Kingsnorth calls the machine - at its core the concept is that prickles can replace goo [0]. I think this is part of why the "getting left behind" language is a catalyst, because some people hear it and think, "good, I would like to be left behind". What will remain to be seen is how things play out for the machine itself. Yes, we are intertwined with its fate, but we don't have to be completely enthralled to it.

Tied to this, I see so much claudeslop in my day-to-day that I have started policing myself for it, almost constantly. I am really sad and annoyed about that fact. Yes, language is a pattern and is arbitrary in a sense, but having it stamped into conformity feels like newspeak. This is only one aspect of a larger "everything faster, better" impulse that cannot continue.

[0] - https://www.youtube.com/watch?v=D4vHnM8WPvU


Everything is Prickles and Goo [0]

[0] - https://www.youtube.com/watch?v=D4vHnM8WPvU


My nontechnical co-workers see the Microsoft 365 outage, the FAA ground stop, and things like this as one interconnected problem. They aren't entirely wrong, but the general fear is hard to dismiss

They are at least interconnected in the sense that we are struggling societally to do big, hard things that we could do a decade ago.

Roman citizen wondering why the aqueduct isn’t working kind of thing.


Interestingly enough it took about four generations, a century, for someone to go back in and investigate the aqueducts properly. I think a lot of this is due to previously easy, simple things being ignored, in part because they are considered menial. The basics of maintenance are easily forgotten.

> The basics of maintenance are easily forgotten.

Also undervalued by society as a whole and politicians especially.

They'd rather allocate funding for a new bridge they can point at and say "I did that" than properly fund the unsexy work of making sure the existing bridges are maintained.

Now apply that across most western democracies, repeat for decades and we are where we are.

The countries doing their vast infrastructure build outs will likely repeat the pattern in a few decades.


The basics of maintenance are... easily cut by VP Chadson Chadenberg, former McKinsey, in his cocaine-fueled spree to increase profitability, you mean?

Anybody. Regular people do it too in their own homes!

The other issue though is that with 24/7 connectivity, any unplanned downtime can feel like a major outage. 50 years ago there were lots of outages, to, but people didn't care as much, they just went outside.

50 years ago, an outage in Seattle didn't affect me in St. Louis as a rule. I'd go to the bank, hand the teller a check, they'd eyeball it, maybe hand me some cash, and process all the things on paper.

When banks started moving to computerized systems, maybe a bank would be unable to process something... but not multiple banks. I can't recall any "outages" that affected banks nationwide for any period of time. Walk into your local bank and find out they can't do something because their big system at the mothership is down? Perhaps.

Consolidation likely plays a part in this as well. I can't speak to the FAA/ATC systems 50 years ago, I have no recollection of what they were like then, but the only nationwide system I can think of around that era was Ma Bell. And I don't honestly recall any nationwide outages back then. Regional, sure. You might have a problem with part of the country getting calls if there was some kind of emergency... (e.g., everybody trying to call their grandmas in Kansas after reports of a tornado) but I don't remember there being any nation-wide problems like that.

We had to count on actual snow for snowdays. :-)


The manual processes don't exist anymore. If the IT is down, nothing can happen.

> And I don't honestly recall any nationwide outages back then

There was this one in 1990 [1], but it's notable because the system was otherwise very reliable. Also, this report says only 50% of call attempts failed, presumably most people would try their call a few times and likely get through.

[1] https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...


I would also say that there are more single points of failure on the internet than there were 20 years ago.

In 2006, Amazon had just launched this new storage service called "S3." It would have struck people in 2006 as bizarre to think that AWS (and Cloudflare, and GCP, and friends) would one day be so crucial to the internet that a bug there could take down dozens or hundreds of companies.


And for a lot of companies (I know this is true for Capital One), the sole redundancy plan for the vast majority of services when US-East-1 fails is to fail everything you can over to a different AWS region.

I'm not surprised. The same type of plan can cover fedramp certification.

That was a frequent question asked at the annual CEO-on-a-stage-in-front-of-the-whole-company event, whether we could be cloud vendor agnostic and setup for an easy lift and shift to another cloud provider, and every year we'd get an answer about how great AWS vendor-lock was for us.

At this point I'm pretty sure 99% of businesses have looked at whether to be ready for a lift and shift between vendors or whether to cut this month's compute budget by using more provider specific services, and pretty much all of them have gone with going with more provider specific services.


Fifty years ago the store owner just wrote down your name and you could pay the bill next week.

One of the coolest things I've seen, system design wise, was working at libraries when the tech/power is out for a given period.

Every service except digital lending/materials continued to function, and there were fallback processes for recording everything that happened so the information could be added back into the digital system once the tech was fixed/power was restored. Specifically designed forms that were easy to use and easy to scan, an exact understanding of what needed to be recorded and why and where it would go later, etc. And a staff of people who were meticulous enough to actually follow through.


This makes sense given that so much of how information theory and data processing works were influenced by libraries, which have existed for thousands of years

50 years ago was 1976. Basically nobody had a computer at home. "Outages" were not a thing.

50 years ago my parents were getting power outages on the somewhat regular. I have had two in the last decade.

I had one last week. And multiple each winter. Its highly environment specific. Unless you're saying you still live in their house?

Very much region dependent. I grew up in Ireland and currently live in the UK in a city. Power outages are extremely rare in both.

Power outages rare in Ireland? Absolutely not. Unless by rare you mean a few per year.

I had one this morning. I live in the middle of a metro area of half a million people.

PG&E I presume? We get multiple per month at this point

We still had a banking system, we still had phone lines, etc. But yes, the general lack of an expectation of always-on connectivity meant that service interruptions were much less of a big deal.

Anyone who hasn't embraced the 24/7 connectivity has been priced out.

That isn’t something that has changed in the last 10 years.

I originally missed OP's comment about "a decade ago", but I don't think our "ability to get big things done" has changed in the last 10 years, nor the number of major outages.

When I hear people talk about "we can't do big things anymore" in the US, they are usually referring to mid 20th century stuff a la https://patrickcollison.com/fast, not stuff that we did 10 years ago.


The topic of TFA and this thread is that these major outages appear to be happening more frequently this year - or in the last month - than we've seen in the past.

I do not believe the FAA has ever grounded the entire country due to a network upgrade going bad. So that is possibly an example of a hard thing we could do a decade ago, that we can't do now.


It may appear, but it's not.

> I do not believe the FAA has ever grounded the entire country due to a network upgrade going bad.

In 2023 the FAA grounded the entire country due to a faulty database update. So yeah, these things happen, people, like you, just forget about them.


Since 9/11, there have been two nationwide ground stops that lasted more than an hour. Both have happened within the past three years. If we get a third soon, we might well have a trend.

Deciding on the ground stop instead of some other type of quick rollback might've been more to do with furthering financial agendas by making a scene (under the guise of "an abundance of caution" which does seem like a more popular thing lately than decades ago) as opposed to it actually being the only reasonable action.

No, that did not happen because every communication related to this is recorded, and will be audited in an investigation, and this would have been known to everyone ahead of time. Most likely the NSTB will investigate, though this has not been announced. There are also going to be congressional hearings and career civil-servants will be subpoenaed.

It doesn't seem hard to use two angle of attack sensors if you pitch the nose of a plane down based on its output.

The thing done today weren't even computationally possible a decade ago. The bar for competitive software is rising exponentially.

> The thing done today weren't even computationally possible a decade ago.

Not the things that are breaking, no. The FAA was doing ATC a decade ago. We had email and cloud servers a decade ago. We had bank websites a decade ago. The scale of those things today is perhaps larger than a decade ago, but not enormously larger; indeed, the increase in scale in those things is much smaller than the increase in computing power in the world from a decade ago. So these things should have gotten more reliable--but they haven't.


Perhaps we would be better off if businesses simply couldn't get this big.

we skill-squeezed ourselves as an industry.

we made these systems simpler to maintain, to reduce the amount of skill we need to pay for to maintain them. And that does reduce the maintenance costs, at the cost of more compute. but who cares about compute costs where compute per unit of money doubles every 18 months, amirite?

unfortunately it also increases the competence required to change these systems without degrading their performance and usability.

since no one wants to pay that extra cost either, we let the systems degrade a little bit at every change instead, hoping they degrade slower than hardware improves.

that bet was bound to lose eventually, because of laws of physics, and because that degradation pattern affects hardware too.

and the double whammy is - with little demand for expertise - the supply is effectively gone too. so now that the walls have closed in, there is no way out.


We also had this decentralized network which turned into a monopolized shithole that fries normies brains. Although that was probably more than a decade ago.

Important stuff is the same. Banking doesn’t need fancy stuff

Do you think someone could take a look at the things you do for work and accurately say whether it needs "fancy stuff?"

Both the FAA and the Microsoft services that failed have been around longer than a decade

I'm pretty sure I was able to do online banking a decade ago. USAA even had working 2FA at that time.

> The thing done today weren't even computationally possible a decade ago

I was able to log into my bank's website in the late 90s, FWIW. Yet today I can't seem to do that.


No we definitely had banking software in 2016.

We definitely did. And stuff like this happened back then too [1]. And even worse stuff before then [2].

[1]https://www.fintechfutures.com/baas/bt-fault-hits-uk-banks-d...

[2]https://abcnews.com/US/hackers-block-us-banks-online-service...


"Nothing existed before I was an adult" type vibes.

AI revisionism is out of control generally tbh lol

I've been working in software for 30 years. This is the largest collection of systems to be down at the same time in that period. I have been literally experiencing a degradation in online services over the last year or two. Its either poor quality AI generated code, or your generation is just incompetent. Your call...

If I were you (or was working at somewhere with an outage), even if it wasn't AI, I would still be blaming AI. Because the alternative means you shouldn't have a job in tech.


that's not really a good characterization. we have known this was a house of cards for decades. someone just turned on the fan.

Where's Agrippa when you need him as aedile?

[flagged]

What do people mean when they say this?

They mean there is a large multi-decade conspiracy theory which ties many world events into one cohesive narrative which culminates in a large scale planned destruction of western power structures wherein that controlled demolition scares the proletariat into being accepting of a new world order in which the global currencies are consolidated to a singular digital currency. This shift basically marks the end of individual freedom since nothing short of a uncontrolled global collapse could ever unseat the hedgimony it would create.

How could the CBDCs possibly be worse than EMV already is? The world of social credit scores and political debanking exists, unless you are buying coke off the darknet with monero.

But for everything else you need, food water shelter healthcare, you need to submit to europay visa mastercard


Yeah, it mostly exists now... And gets closer to the end state every year.

ending of fight club?

Well, digital centralisation and interdependency without fault hardening is a big problem. Just because it presents itself in slightly different ways across industries doesn't mean it is not a fundamental issue in our society.

I don't see much of a problem with depending on AWS or Google Cloud. It's Microsoft who are particularly flaky. And always will be, because IT people choose them regardless.

You are still "centralizing" on "someone else's computer". Just because those two have not, yet, had a MS365 event does not mean they won't have one at some point. And everyone "centralized" there will have a bad day when that happens.

AWS has had them, and is passed the growing pains. At this point when/if AWS goes down, it will still have a far higher uptime than others could manage, let alone those going it alone.

Microsoft on the other hand, has proven that it will never be reliable.


AWS quite literally lost an entire region in the Middle East and has said they will not be able to restore it.

You can argue that this is a result of something outside of their control, and you would not be wrong, but if the point here is about centralization rather than simply a particular company's operational procedures, then direct strikes on datacenters should be part of that conversation.

I was an Amazon red badge in the EC2 org, fwiw, and do think AWS has excellent operational culture, far beyond what I saw while working in the M365 org, and beyond what I've seen at other tech companies and hedge funds where I've worked. But that doesn't obviate issues with centralization.


Centralization is almost a good thing for extreme cases though.

If Bank of America's central datacenter (cloud or on-prem) gets hit by a cluster bomb, taking out banking for ~half the US, you can be damn sure that very smart people are going to be working 24/7 until it's resolved.

If you have all your savings stored with a local credit union, say Seattle Credit Union which has 50,000 customers, and they get hit with a ransomware attack... well, good luck. You (probably) won't lose any actual money thanks to FDIC rules, but it might be weeks or months before their services are back online.

Similar to self-hosting vs cloud, you get a bit of a paradox where it's overall better for society if there's no interdependencies so a SPoF outage doesn't impact everyone, but things are much worse for an individual in the event a localized outage does occur due to economies of scale, or lack thereof in this case.


Anyone who says "someone else's computer" is just giving away that they don't understand the difference between some pet vm and a proper cloud infra.

One team won’t win the cup every year, the issue here is that everyone has picked their favorite while ignoring the risks of the league.

Have you tried using Azure? (I have, it is not good)

It’s a tragedy that much of European industry and public sector are so deeply embedded with Microsoft that they barely even consider other options, even when everything Microsoft touches seems terrible

This is also true in Oceania (especially NZ) in my experience. Companies here are much more embedded with Microsoft than companies I worked with back in the US and are less willing to consider other options. My company actually does most stuff with on prem windows servers and a bit with Azure. One aspect for NZ in particular is that there was no NZ region for Azure or AWS until 2025 and parts of the country are 70+ms from Sydney, so for anything where latency is significant, on prem stuck around here a lot later.

I feel like on-prem Microsoft stack is a very well known and controllable beast.

Its pros/cons are known and gaps can be addressed with 3p software. You just keep on old outdated, but proven version of software and keep the perimeter locked tightly.

its the azure cloud that is the wildcard.


GP isn't talking about on-prem I believe. Small/medium businesses in Europe are much more dependent on MS than their US counterparts. Alternatives are mostly out-of-scope from their point of view.

Don't forget that MS has the advantage of non-English languages support that many alternatives lack.


> I feel like on-prem Microsoft stack is a very well known and controllable beast.

I feel like it is very unknown, there is no documentation and everyone relies on comments on random forums that are now 21 years old which tells to set undocumented registry keys to undocumented values.


I don't understand why people don't prefer to edit text files at arbitrary locations.

> It’s a tragedy that much of European industry and public sector are so deeply embedded with Microsoft

It's not just Europe! :(


How quickly we seem to have forgotten how often aws us east went down.

They’re expensive and centralized. I always thought the general wisdom was to bootstrap your business with serverless infrastructure and then eventually work your way back to an on-prem-first philosophy. I guess a lot of companies skipped that last step to…save…something?

Yeah the FAA didn't see a problem with a single point of failure either look how that went. It's not flaky until it is.

>They aren't entirely wrong,

They are entirely wrong.

None of these outages, as far as we know, have a common cause. They just happened by coincidence inside the same news cycle.

Nor is there any evidence to blame them on AI, or tech layoffs, or hackers from Iran. It's more likely something boring and technical that doesn't fit into anyone's narrative.


My guess is a total disrespect for actually qualified people. You know, the ones that specialized in one thing, for their whole life. Every company tries to see how few people they can have and still run fine. And move fast. And no trainings. And no recognition for the people that could actually train other people. I think all these come together at some point and you get multiple catastrophic events in multiple industries.

If you have no skills it is hard to evaluate if someone you intended to hire has some.

Nailed it. Downward pressure on salaries, layoffs, outsourcing all drive technical excellence out of an organization. Then, when companies DO hire it's very hard to find competent people (and I mean find them at all, not find them at the cutrate you want to hire them at) and the people you do hire are just seat warmers

I would ask a few questions:

1. Does there seem to be a larger concentration of outages recently? (I would say yes.)

2. Is there actually a larger concentration of outages recently, or does it only seem that way because of an increased focus on this? (I would say that there is really a larger concentration, but I'm less certain about this.)

3. If there is actually a larger concentration of outages recently, is that because of AI? (This is the point at which I become pretty unsure either way.)

4. If it is because of AI, how exactly is AI causing this? (I think more widespread vulnerability discovery / exploitation is more likely than "everyone is vibecoding everything now and that's screwing everything up", but I'm not entirely sure.)

5. Is this just transitional growing pains, or is this a new normal, or is this the beginning phase of a pattern that is only going to accelerate? (I don't know the answer to this one at all! I hope it's the first thing, but I wish I were more confident.)


> They just happened by coincidence inside the same news cycle ... to blame them on AI, or tech layoffs, or hackers from Iran

I will add myself as another datapoint, yesterday, I investigated my first ever reflection amplification attack that was bringing an entire network down, the ips originated from both china and iran. While running my oss work: https://github.com/mickael-kerjean/filestash, over the last 2 weeks, I have received about 20 security reports when the usual rate is once every few months, and on top of that literally every single form coming on my website is being abused by spam, the rate of spam message coming through there has 100x increase over the last few weeks. I have never seen that much security work pushed over me ever


What evidence do you have that staff reduction is not at the root of some of this?

Sorry, are you asking them to prove a negative? They said there was no evidence it was.

Or maybe the more evidence there is that's unrelated to staff reduction, the less likely it is a big factor?

It could be some people rushing to get code into prod before the holiday freeze, or just a coincidence.

Either way, now that AI is more capable, you can really mess stuff up at scale.


The messing stuff up at scale is more because LLMs aren't capable, and people treat them as though they are.

Is this how the "AI will lead to the end of the world" type stuff I've been hearing about starts?

Code freezes for the holidays were several weeks ago. The code would have been deployed weeks ago too. And if AI writes better code than you, that says more about you than AI.

Which company are you talking about?

No, it all comes down to the decision-makers at the top being bad at their jobs.

One way or another, they've tried to avoid addressing tech debt, avoid hiring and retaining good people, and have more or less been kicking the can down the road / running on the edge without knowing it.

Or else they did embrace AI-driven "move fast and break things" approaches, and so they broke things.

They flip-flop between maximizing profits and draining operational divisions of any resources, and hoping to sprinkle a bunch of IBM magic pixie dust all over their infrastructure to solve all their problems without needing to understand any of it. You rarely find one that strives to understand problems deeply and invests in long-term planning and maintenance before a crisis hits.

The result is old systems that eventually break, system upgrades that go bad because nobody understands them, security holes that someone might have hacked into, etc. All the same basic root cause at the top, which is that most of our leadership class is full of incompetent short-sighted narcissists. And it "works" because most of the time it pumps the stock due to lower costs, and we only see the fraction of times that it causes headline-grabbing catastrophes. On a long enough timeline, though, they'll happily drive our entire society off a cliff--and we're getting closer and closer to that (at least politically).


Even moreso, it might also be that responsibility (sic Liability) is applied asymmetrically AWAY from the decision-makers you reference.

They reap all rewards and bear none of the responsibility beyond saying verbally “I accept responsibility” (usually in the form of more layoffs and firings for those more explicitly involved in implementation). How many C-Suite executives will be shown the door as a result of poor staffing and strategy that lead up to these various outages?

Conversely, when outcomes are positive, compensation and notoriety are disproportionately attributed to said decision-makers.


I sincerely doubt it is the IBM Pixie Dust and Mainframes these banks use that is the root cause. The mainframe systems programming teams at places like JP Morgan are supremely competent. I've worked with them many times.

The app developers hooking that all up with the cloud? Sometimes less so.

I suppose we will just have to see what the root cause is, but I'd put a bet on it not being down to "old IBM Pixie Dust nobody understands" or anything of the sort, and suspect it is cloud related due to this occurring across multiple banks.

In outages I've gone through root cause analysis for that involved mainframes and cloud and just general hybrid cloud stuff (this is my specialization as a mainframe systems programmer and consultant on cloud integration), it's almost always the cloud that was flaky. Mainframes are actually understood quite well by the priestly class that is trained on them.


Sorry, I think I wooshed you on that.

IBM Magic Pixie Dust ad:

https://www.youtube.com/watch?v=3nbEeU2dRBg

I was trying to allude to LLMs/AI as the IBM Magic Pixie Dust.


Ahhh gotcha, fair enough haha. Wasn't aware of the ad.

The thing is, we don't know.

You saying they *are* entirely wrong is as baseless as their claim. Sure, you can say they don't have a common cause, and I agree with that, but chances of these events with different causes occurring so close together is very low and the chances of a same root cause is probably more plausible.


As history has shown again and again, blaming something only requires political will, not evidence.

I am assuming you don't have a direct line to each of these outages, and you have no proof that there is no common cause. A common cause could also be something like "poor production deployment practices" or "reliance on AI built software", in which case...maybe they aren't entirely not wrong.

It's not my job to prove they aren't related. That's the default; most events are not related.

It's your job to provide evidence they are related, and you have nothing except speculation.


AI has been finding new critical 0 days at an increasing rate so it's actually not unreasonable to think these might be related.

Yes, but you have no proof while saying someone is for certain wrong. Seems like the burden is on you to back that up /shrug

It’s actually hilarious that you made the irrefutable claim, and then you are telling me to find proof.


The boring news is that laying off 100th of thousands of IT workers because of a newly promised, never-seen-before technology, does indeed cause trouble. Who'd thought that these people did in fact create value?

The value is created by maximizing your screentime to show you ads that sell you stuff that you don't need.

Genuinely, what would it take for you to consider that it could, in fact, be the AI push?

I'd want to read the post-mortem analysis from the team that diagnosed and resolved the fault.

They'd really be the only people with information about the cause of the issue; if they blame an AI-generated code change or the actions of an AI agent, that would be good evidence.


Not trying to be that guy, but they do have a common cause: networked compute.

There are many more systems in the world outside of it, even if it is relatively ubiquitous at this point.


Where there's smoke...

"As far as we know" just indicates a lack of data, not evidence that they're not related. This is especially relevant for a political administration whose response to COVID was to bury their head in the sand by hiding data. Also: A public comment of "AI is hacking all our shit" wouldn't exactly calm the public.

It's fine to say, "I'll wait until we know more", but in the meantime AI progress continues and the sheer volume of cyber vulnerabilities it's uncovering is undeniable. It's not surprising that some state-sponsored bad actors would be use this as an opportunity to infiltrate too...


Seems to me that most state sponsored bad actors would want to infiltrate in a less noisy way.

You also don't have any evidence that they are related.

In the absence of data, you are assigning blame to things you already hate, like AI or the administration. Motivated reasoning is highly prone to error, and it's a form of stupid-brained thinking.


No blame or hate. Just hypotheses with conditional probabilities that will be updated in response to more data...

It's extremely funny how many of your replies are proving your point exactly. One reply says "Yours is easy to hold since it can't have evidence" when it is simply stating that _there is no evidence of a claim_. Imagine that, "There's no proof of X" is somehow a magically unfair argument because it can never be proven! Wow! Nobody has ever thought of this before. Guess we have to assume there's inherently merit to X, then, and don't have to worry about actually constructing a proof of it (that requires patience and knowledge and first-party sources and does not get you internet points).

As a technical person, I tend to find it hard to dismiss as well. What's easier to explain. Someone said it in a comment, someone keeps pushing for the cheaper solution cloud infra was cheaper and now we see the cost of that.

I swear nothing works fine anymore, the world feels broken. Maybe those particular incidents are not interconnected directly but they are connected in the ether in the sense that everything is incredibly complicated and precarious.

See the work of Joseph Tainter [0] - we are in a living system that has grown drastically with the input of more energy. We're hitting a threshold where the metabolic cost of maintaining our current level of complexity matches or exceeds the in-drawn energy from outside the superorganism, so catabolysis is starting. See also Nate Hagens [1], Daniel Schmachtenberger [2]. Good news is that this is part of the adaptive cycle [3]

[0]- https://risk.princeton.edu/img/Historical_Collapse_Resources...

[1] - https://www.youtube.com/watch?v=h5VWZm7ESfk

[2] - https://www.youtube.com/watch?v=KCSsKV5F4xc

[3] - https://www.youtube.com/watch?v=5WPB2u8EzL8


It's cultural. The current culture is "see no evil, hear no evil, speak no evil" adjacent which inevitably leads to no one pointing at the giant pink elephant in the corner (misuse and misapplication of AI).

None of the responses to your comment get to the heart of the issue. People believe these are likely interconnected because they no longer trust the authorities telling us they are not connected. I certainly don't trust the government, and the mainstream media seems unable to accurately cover such issues.

To me this is the key issue of every "conspiracy theory" regardless of the likelihood of its veracity, right up to the flat earthers.

These theories are the product of a society in which every major institution is corrupt and untrustworthy as a norm.


they're interconnected in that we're all getting really overloaded by the workload created by AI, and even though we can use AI to get through the workload as well, we humans still have to be aware of everything flying by, and I think things are falling through the cracks more often. These days I work on ten or twenty issues at a time. it's a little nuts

It is an interconnected problem, but it's the problem we're all already aware of. All these businesses and the FAA have gutted their operations teams. There's too much to do, and AI doesn't take less time.

It’s natural to turn to AI based development as one of the causes of the increase in large scale outages.

I prefer to ask a different question. Will all this AI development delivering us amazing new software? Are we just going to get 1000 free clones of Adobe Photoshop or can we expect some really exciting groundbreaking new tools? Perhaps all the really clever use of AI development is in the pursuit of better AI and if that’s the case, why?


Yeah, I'd probably extend it to software too: which software actually delivers real value to society, and which is only created for software's own sake.

As to why, well economy must grow, no matter what.


It boils down to leadership. If all C-suite asks for is "more, faster and cheaper" then enshitiffication gets worse.

Unfortunately it will take many more outages for folks to realize that serious engineering talent has great value in the AI age. It's their judgement that helps speeding headfirst into building unreliable unmaintainable and difficult to extend systems.


turns out AI thinks like an engineer an engineers are mostly not that innovative.

innovative on which aspect? I want a database that works that is really fast but don't innovate in the area of losing transactions and making up transaction IDs and amounts

software quality is going down in general

not sure if vibecoding is a cause or consequence, but it’s related, it requires as an axiom to stop caring about internals. And it seems to require disregard for quality too.


Basic competency is going down. Everything is outsourced such that no one knows how their critical systems work.

Competency going down, timelines being rushed, project managers and CTOs forcing new tools at breakneck speeds.

i'm all for experimentation and adding new tools when they're shown to actually help, but it seems like large software companies (especially banks and e-commerce) just don't care about quality anymore.


I think quality of most products is seemingly going down. To me it is a likely consequence consequence of increased financialization and money being expensive.

Also of the increased monopilizations / duopolization as well as increased abuse of network effects in many areas that removes bargaining power from consumers.

I worry that this isn't the right metric for diagnosing whether there's a problem. I'd argue that we've had instances in the past where a massive increase in how accessible programming is ended up being a net gain in the long run for software even if the average quality went down (e.g., having computers move from giant room-sized mainframes that only companies and governments could afford to sitting in the living room on a table probably reduced the average quality because untrained amateurs could start write software even if they weren't skilled enough to be employed). I could potentially be convinced that this isn't the same and that it will end up being worse for software in the long run with a stronger argument, but "software quality is going down in general" isn't convincing enough evidence for me.

All of this is also downstream of what I think is even more important, which is what's better for society in general. Obviously low quality software is not better for society than high quality in a vaccuum, but I also don't think there's a single obvious objective answer for whether the influence of software on society is a net positive or negative today compared to, for example, when computers were all mainframes and all software was produced by people paid to write it. There are a lot of differences in society today compared to back then, and software certainly has some role in how we ended up here, but trying to attribute exactly what and how much is not an exact science.

This is a lot of words to basically say: if you're confident that this is a problem, I think it's important to state exactly why, since I think the interest parts of the discussion are in how you've concluded that.


Contrary to "vibecoding" (which is increasingly becoming how anyone programs) being a cause or consequence, it'll probably be part of the solution.

> Contrary to "vibecoding" (which is increasingly becoming how anyone programs)

Commonly asserted but not true. Most programmers are not just letting the model take the wheel.

> it'll probably be part of the solution.

LLMs can't solve the slop problem they themselves are creating.


> Commonly asserted but not true. Most programmers are not just letting the model take the wheel.

In what way did you interpret vibecoding as taking the wheel away from programmers? It certainly does in writing code. Generally developers are still involved in architecture design and defining requirements to give their agents.

> LLMs can't solve the slop problem they themselves are creating.

Human slop is the real problem that LLMs can and are solving.


>Generally developers are still involved in architecture design and defining requirements to give their agents.

What do you call someone offload all of this to the agent too ?

Yup, it's called vibecoding. You don't care how agent achieve it, you care only about the result.


Software quality peaked between XP SP2 and the victory in the market of node.

Very well put, your comment alone should probably be a dedicated post in here with people commenting on that timeline (people who were actually programming back then).

Maybe it was because I was a lot younger back then and a lot more idealistic, but especially for the ~2005 (i.e. just after the dot-com crash) - 2010 period (just before money would ruin everything in this industry) I can say that the programming bar was placed a lot, a lot higher compared to where I see it placed now. Software wasn't perfect back then, of course, but there was a real push (belief, maybe?) that with enough will and with enough new-knowledge-being-acquired then we (meaning all of us as an industry) will make things a lot better. In part the existence of this very forum is a simptom of that philosophy, what if it (originally, maybe it still is) being written in Arc, or Reddit originally being launched in Lisp.


Maybe node software was good initially, but its model was unsustainable, it relied on outsourcing to volunteer programmers, relying on such programmers means increased vulnerabilities, supply chain incidents, and low quality of software.

Vibecoding is an extension of the culture of overrelying on massive external dependency graphs. It's a preference for visual confirmation that something works and a disinterest in how it works. I think it's also related to the tendency of content feeds that are based on likes and swipes instead of searches, instead of specifying what we want, we are reduced to approving or dissaproving (until we are removed from the loop entirely)


Here's my “favorite” chart showing the steady decline in software quality over the last 25 years: https://trends.google.com/explore?q=%22something%20went%20wr...

That's more like the rise of Google (which may or may not be correlated to software quality), as "something went wrong" is an error message popularized by them.

considering how the experience with most SaaS products right now is insufferable with broken states and UI, I imagine the same broken behavior is leaking in every other area of product development, so I agree

Source = trust me bro. What’s that bias where things are so rare that when they happen it makes news, so you think they’re very common (eg plane crashes?).

Uptime and quality of software is about 10000x better than it was 10 years ago. Things just work all the time now.


> Uptime and quality of software is about 10000x better than it was 10 years ago

Extraordinary claims require extraordinary evidence. You're being the 'trust me bro' in this situation.


I'd argue that just illustrates the point that it's impossible to have an interesting discussion without a stronger claim from the parent comment. If I say something that's too vague to falsify, and you can just as plausibly claim the opposite, you're not at fault for failing to turn my claim into a more rigorous one.

Also, source = trust me bro? How can you make the opposite claim without backing it up either

I have a predication: In the next few years having any connection to the open Internet will come with massive risks. I.e. if you're not a government, military or $$$ company you won't be able to afford the steps it would take to harden said system against the onslaught from AI attacks.

Some other thoughts:

* Will the Internet become balkanised?

* These smaller networks will require a lot of personal information to even gain access.

* The return of the Sneakernet?


It's entirely reasonable to write secure internet-connected software as long as you limit the scope. E.g., the web doesn't have to require each client to bring an insanely complex programming environment. 00s level of HTML and CSS was good enough for most use cases actually - certainly better to go back to that than to give up on the internet entirely.

Similarly, if you don't insist on moving so fast you can actually fix bugs and create hardened systems.

And finally, we can also give up on the notion that the Internet needs to be secured using entirely technological means. We don't have that requirement for anything in the physical world or even for other communication channels (e.g. phones, mail). We don't need to make murder literally impossible and we don't need to make computers literally unhackable. Doing so is a choice with a non-zero cost and if the cost increases dramatically we need to re-evaluate the choice and reconsider alternatives.


I'm as technical as most of us ... and I'm lightly suspicious ...

Not the same root cause of course, but there's something in the water


The aura is definitely misaligned and our collective chakra is waning. My earth stone are vibrating in a concerning manner

I mean, from memory, we didn't use to have this many outages and especially not this severe, this close together.

If everyone cuts the bullshit, we'd admit that save for edge cases, the industry is going off the rails thanks to AI and its promises of jewels. The tailpipe is out of exhaust and people are still huffing. Of course shit is falling apart right now. We're living in a state of utter delusion about a tool responsible for helping us build and maintain systems.

The wise among us will take caution, but I think this was a tremor, not the actual quake.


Internetworking

I think you mean internotworking.

Any reason why it isn't the case?

Because there’s not one, shared cause between the outages. The lay person claims it must be AI, but that’s a bit meaningless.

The reason I say not entirely, is because there is kind of a shared cause. One way of looking at is is that the root cause is complexity, optimized for efficiency and speed rather than robustness. It's a feature of the economic superorganism we are operating as part of. See Joseph Tainter on Collapse of Complex Societies, the work of Nate Hagens, Schmachtenberger & Liv Boree talking about Moloch, etc.

Or put a bit more cynically, it is a symptom of a society-wide drift towards lower accountability and duty of care. It's a type of tragedy of the commons. Selfishness and externalized cost/risk means nobody optimizes for systemic reliability.

People "vibing" everything is just another component of this, not the root cause. Using AI tooling is just the new technique for what used to be called phoning it in, or going through the motions, or faking it until you make it.

Unfortunately, it's nearly always been an effective tactic for a selfish few. It's problematic when we don't have an equilibrium that keeps it in the minority...


Yes, that's how you take separate causes and transmorgify them into one omnicause, by muddying the waters until you can't see the differences anymore. Don't do that! It just adds confusion.

(The same thing happened with separate company's problems becoming "Big Tech." It didn't help.)


Once you give separate things a shared name they become the same thing. We could call those in the claim to be "race to the bottom effect" or "incentives against resiliance". The water is no longer muddled, its now a slushie.

You could call it "category theory" or something.

Snarky, but there's a point: Abstraction leads to viewing different things as the same thing. They are - at one level of abstraction. And they aren't at other levels of abstraction. It's not a problem to have both viewpoints.

The trick is that, when someone blends things together, to not lose the fact that they are also separate (at other levels of abstraction).

So I have no problem saying that "optimizing for efficiency and speed" is the common cause of all of these problems, and saying that they all have different (immediate) causes. Both can be true.


Sometimes these abstractions are useful for design or analysis, but maybe not for explaining them to others? A library or programming language will probably be more usable if its users don't need to know any category theory.

I'm not sure "optimizing for speed and efficiency" can count as a common cause, though, because it's not a single event. It's more like a set of events that have something in common.


Is there?

Apparently the FAA outage was caused by the pressure to do things quickly, fueled by the AI urgency.

It feels like all the IT industry is feeling this pressure.


Maybe we're not looking far enough up the chain. Everyone used to use forums and other websites run by other people (eg Substack) to read and discuss about things they don't understand.

Now when people have a question, they ask AI, which is wrong a lot of the time.


What does the "pro" person claim? That it's Iran?

That there has never been an incentive to build quality software. As a result we spent 50 years building our society on top of poor quality software. We're paying for it now and will continue paying for it until the incentives change and quality becomes a focus.

I suspect there are incentives for Iran not to be identified as the culprit right now whether they are or not. If it is indeed Iran that would show they have the capability to do this and maybe the administration doesn't want that to get widely circulated. (and if it is a state actor it doesn't have to only be Iran - could be a coordinated effort - NK just did the big Binance heist and Russia can do this kind of stuff too)

NK's heist was against Binance, not Coinbase.

The main incentive for any nation state to avoid attribution to direct and material harm to many Americans is that they risk awakening American bloodlust. Americans are unhappy about the current Iran situation and wish it didn't start and that it would end already, but sufficient Iranian escalation would turn that around real quick and suddenly there are boots on the ground among other things.


From the other direction I could see that the administration wouldn't want to attribute incidents like this to Iran because it would just make Americans more unhappy that the administration decided to poke the hornet's nest when they really didn't need to. And like with not anticipating the closing of the SoH, they completely dropped the ball on cyberattacks as well.

I've heard claims that it's the people trying to get the US to fight Iran, recently unsuccessfully. Mossad is known for being very skilled. Not saying it necessarily is, but it goes back into the distrust of government angle. Trump saying he's fighting this war in Part for Israel doesn't help.

That it’s a nuanced mess of factors. Like reality always is.

To be fair, have there been post-mortems for any of them? Besides vague tweets or press statements?

https://www.365fanatic.com/wp-content/uploads/2026/11/PostIn...

That's the preliminary post mortem from Microsoft.

I don't think we'll ever get anything honest out of the administration for the FAA outage unless Congress really wants to dig into it and make the results public.


Oh wow. Time for a "latent bug" drinking game?

> The lay person claims it must be AI, but that’s a bit meaningless.

Must be the water then


Imagine a tree of relationships branching up to the very top labelled, "Because the Universe Exists" as being the root issue that relates everything.

Saying it's not related applies an assumption of what level of specificity you are thinking about. And that's how you quickly de-sync with other people, especially "lay people" who look at you funny when you claim there's just no way these things are related when it's hard to believe that based on experience alone.

As other comments suggest, it probably is related to an extent. Related by complexity of our Internet-based systems, related by age, related maybe even by enshittification of expertise, etc.

The further up the tree you find these relationships, the less specifically actionable they are (you cannot CVE enshittification), and the more they are about "what are the overall social/cultural/human issues we're grappling with in this era?"


Aviation got a whole lot safer when we started looking at the systems causes instead of the specific ways individual people, airlines, manufacturers screwed up. Ditto for anesthesiology, cars, etc.

Sure, focusing on individual small causes is helpful too, and it's a lot harder to do the higher and higher level system stuff. But, it can be done.

edit, P.S.: It's worth keeping a bare level of suspicions that there could be a common lower level cause, whether through inadvertent dependencies added or through hostile action.


a) this is not known, there is no reliable data on either the FAA or new bank issue; and the information forthcoming from Microsoft may obscure primary causation

b) there is also a risk of sophistry and language games obscuring common sense truth

In the world where a state actor is engaging in active sabotage as part of a cyberwarfare effort, there is one shared cause.

Which exploits are responsible in each case may differ. Still one cause.

Do I think it's Iran?

No idea.

But the point is, none of us here have any idea at present.


I can see why people think it. I don’t think I remember a 7-day stretch like this…. ever?

- Massive Microsoft outage - FAA ground stop - Now this outage

Definitely a black swan at least? It’s not even the number of failures, it’s the scale.


Also last week, a hack of Nat'l Weather services web site (but not affecting services) and a minor outage at some Amtrak services on the East Coast.

It's true that we don't really know, and my first instincts lean toward hacking by a foreign government, but then I start thinking about media reports of shark attacks every summer. Shark attacks aren't unknown, but certain media channels like to pounce on them for ratings, and after a while to the average person it might seem like Sharknado. Stuff goes down all the time, but in this moment we're just extraordinary conscious of it.

One might argue that lazy sloppy work and the generally pathetic state of IT maintenance is the common cause.

My dad is a doctor. He keeps asking me similar questions.

I told him it's just like how he keeps telling his patients to watch what they eat, drink and smoke less, and get some exercise, but they don't, and then they have a heart attack and wonder why.


If an unfriendly foreign power were attempting to impose on America the insecure and unstable vibe-coded digital infrastructure we’re building today, we’d probably view it as an act of war.

Everything you said has been true for a long time, vibe coded or not. How much of the Internet proved to be exposed to a supply chain attack on node? That wasn't AI, that was just how things worked out.

Or the H-1Bs

Interconnected or not, the events feel like they are. Like Mongolian skirmishes, hit a flank and retreat, Parthian shot, appear out of nowhere elsewhere and do the same and hit a different flank, a bite at a time, cause panic and distrust.

This doesn't seem to happen in other countries (except the fridge hack) and they use the same tech. What are they doing right? Do they beg the Congress for $50B to improve their ATC infra? It's annoying to hear Duffy say "my network", "my routers" give me money. It's OUR network.


Same with my parents, until I told them if FAA comms ran on teams and outlook we would have been boned a long time ago!

Maybe Jensen was what was keeping all of the nation's infrastructure running

Three massive outages in three business days in a row does seem like a pattern.

Or is this a period of every blip being watched more closely and becoming a headline after the recent two?

Dude these are not minor issues that would have flown under the radar in the past either.

We (on HN) as generally technical people (despite the HNers who seem to think you can vibecode a NVidia competitor) that all of these incidents are unrelated with vastly different causes.

But I think there's an increasing consensus among "normies" like your coworkers who see these as related. They don't know why but assuring people they are unrelated fails in messaging terms when it feels related.

I think this will be an increasing problem where so much more code is not only written by AI but reviewed by AI. Things will slip through and companies won't be punished for these outages in terms of public perception (because it'll happen to everyone) or by the government. We just don't have serious consequences for corporate malfeasance or gross negligence. Like AFAICT the Sackler family should all die penniless in 6x8' grey prison cells.

Unfortunately we live in a country where the Sacklers get to stay billionaires while 1 year olds are shot dead at the behest of Walmart over a pack of diapers that weren't even stolen... with zero consequences [1].

It feels like this is just the next level of entshittification.

[1]: https://www.nbcnews.com/news/us-news/mississippi-ag-says-pol...


There does seem to be a correlation with the fall in use of Substack and the frequency of these fuckups.

Not a bit occam razor's fan, eh ?

I’ve had people asking me if it’s Jensen Huang’s doing.

I guess for most “technology” is one big blurry mess.


This is the obvious consequence of (failing to) replace software developers with AI slop. Despite the protests of the Antrhopic shareholders who dominate this site's comments, LLMs are hideously poor at designing software.

My nontechnical co-workers see the Microsoft 365 outage, the FAA ground stop, and things like this as one interconnected problem.

Wait till they hear about United Airlines.


It is interconnected. Everyone bought the "cloud is up 24hrs" bullshit. My company had physical servers and we had 99% uptime because we had competent people. Profits are driving unsafe transitions, such is capitalism. Look at our crumbling old infrastructure and ask yourself why the "digital highway" would be any safer.

Banks, the FAA, Microsoft all have significantly higher uptime than 99%

hahahahhahaha .. no

Because they are. Trump just said that deranged democrat/antifa was cutting cables everywhere.

Overall uptime is better than pre internet 9-5. It's so typical to become so used to technology that any disruption seems bad.

This misses that expectation versus reality is key to how a human experiences something. If it's 20 degrees F in November where I live (MI), I'm going to grumble and get about my life. If it's 20 degrees F in July, I'm going to be freaked out. The same situation (no access to banking outside of business hours) can have radically different context and communicate different things to onlookers. See the former USSR's reaction to seeing Swan Lake on TV in 1991.

The absence of something that once existed is not the same as that thing never having existed. A null entry and a non-existent entry aren't the same.


Yes, we should all stop complaining about unexplained technical interruptions in our infrastructure and go back to being grateful for every minute that my bank's website is open on the weekend.

/s


Honestly, I can't help but hope that ends up leading to enough popular support for more regulation for AI that something gets implemented. On the other hand, there's also ample historical precedent for hoping that a mass movement heading in the right direction ends up stopping at the point that feels reasonable and then being surprised when the momentum continues and people like me looking foolish for thinking that it wouldn't, so I might just be overly optimistic.

AI is not the cause of this kind of problem and it's regulation won't make it go away. Maybe it needs regulation, maybe it doesn't, but you're missing what's happening if you think that's what these widespread failures and vulnerabilities are about.

AI is acting as a sudden precipitator for problems that have been latent and growing for at least 15-20 years.

In pursuit of maximally rapid innovation and reduced capital expenditure, the software industry adopted a "move fast and break things" mentality and homogenized code and service infrastructure over a few winning cloud vendors, SaaS providers, and code solutions/frameworks/packages/etc.

The pitch was that you could rely on downstream experts to do better work than you could, that the biggest vendors and publishers had the best experts, that have them invest the capital would look better on your accounts, and that their scale would give you the best value for your investment.

The caveat is that effectively all technology products have serious faults lurking in them, even with the finest expertise working on them (not that most actually had that), and that when everybody homogenizes over the same providers and solution the industry shifts from a bubbling froth of localized bugs to a heavily correlated house of cards where widely correlated failures manifest in widespread collapse and catastrophe.

Whoops.

AI is acting as a breeze against the house of cards, but lots of things could have and would have and did, especially as it continued to inflate year over year.


If you're saying the parent comment I responded to wasn't talking about AI, what are you saying they were talking about? If you agree they were, why are you trying to correct me instead of them?

From experience, since the parent commenter didn't specify, I would take the "coworkers" as having a vague sense of dread and distrust with no real confident insight to the cause. They're likely to nod along to any explanation they hear from anyone they think is more technical or news-following than themselves, and then maybe repeat whatever last such they heard when chatting. With the occasional exception of politics, that's how most people approach technical topics outside of their expertise. I didn't guess what they might have in mind today at all, and didn't feel the need to.

But you did speak directly to AI, and specifically to advocate for regulation of it, which is why I replied to you. You were being specific and material in a way that the parent commenter was not, and that invited a different response.


I didn't specifically claim that it was the direct cause of this though, so you still decided to confront something I didn't claim over something the parent comment didn't claim

> With the occasional exception of politics, that's how most people approach technical topics outside of their expertise

You give other people way too much credit.


What's actually more likley are power grabs executed while the chaos is attracting all the attention. That's how we ended up with DHS.

I don't really see how regulation for AI helps. Companies were on this crash course of trusting Microsoft (or pick your other dependency that cant be eliminated as it is a monopoly or the minimum) to provide more and more of surrounding infrastructure. Until they had AI to run simulations of outside.competence there wasn't a snowball's chance in hell they were competent enough at IT to run someone else's infrastructure.

Haha ask them how much they'd pay for fewer problems

You should first explain to them how they can know that they are paying for fewer problems and not just higher executive bonuses and shareholder dividends.

Somebody pays you every month presumably knowing that. I understand the thrust of what you are saying but it's funny to me that your response to my simple idea, "people do not pay for shit," is, "lame hyperbole"

It's not "lame hyperbole" that the industry has move to full value extraction maximization and deliberated obfuscated everything so that cusomters don't know what deal they are getting.

My nontechnical co-workers see the Microsoft 365 outage, the FAA ground stop, and things like this as one interconnected problem. They aren't entirely wrong, but the general fear is hard to dismiss

enjoy the ride, at least twice. that's a book that bears rereading

I couldn't finish the first chapter. It read like a cry for help, suicide ideation turned into fever dream. It was deeply uncomfortable, and I've yet to hear a description of later chapters to explain why I should have kept going. Especially now knowing the trajectory of its author.

Fair! It is not for everyone. I view it as a product of postmodern angst, like office space or foght club.

See Tainter's work on the collapse of complex systems, and watch this nice video from B. Sid Smith. [0]

[0] https://www.youtube.com/watch?v=5WPB2u8EzL8


See also Donella Meadows (Limits to Growth) on sustainable systems.[0] News of the book's debunking was greatly exaggerated[1] of course.

Drink every time you hear "and notice how on the graph, everything looks great right before the end!"

[0] https://www.youtube.com/watch?v=hhSpzQhvFS8

[1] https://www.theguardian.com/commentisfree/2014/sep/02/limits...


The future is local, planned or unplanned

bc we accessed oil

History is the process of oil getting into everything. The financial system is a structure for distributing surplus energy founded on growth resulting from accessing a material which can do 500 people's worth of work in a single barrel.

Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: