Are these outages caused by introduced bugs, though, or by load issues?
As someone who has spent many years working in high load environments, this is not an uncommon pattern.
You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.
And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.
This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.
It could be that the extra usage just pushed them past one of those capacity thresholds.
Which is somewhat humorous because it was arguably more stable when they ran on their own hosted colo infra before moving to Azure. This was a choice versus keeping the infra compartmentalized and using Azure for elastic overflow compute needs. I'm sure marketing and bonuses rest on throwing it all on the Azure quicksand though.
Except, lots of devs use github, and see what using Azure means. Management is seeing that downtime too. So what insane person will ever touch Azure willingly?
Hotmail used to run on bsd, and they tried to move it to WinNT, back in the day. Same thing. Disaster. And it was held up as a reason to never use winnt for serious work, for years.
Microsoft just always shoots themselves in the foot.
And also the most important few thousand accounts will get custom support and all sorts of patchwork and MS will put the effort to solve all their problems. So the customers that matter to MS are likely having a great time on azure. And for every ten thousand of us chums that leave azure, there's probably one sweetheart deal won through wine and dine that more than compensates.
Haven't worked in AWS but as a customer of AWS, GCP, and Azure there is a quite a difference between running workloads that are doing long-running computations on AWS (on demand is so rock solid that loosing a box is a rarity, I run on spot and still complete almost all of my runs) and running on Azure's "on-demand" style offering feels like spot (or worse) on AWS.
> on demand is so rock solid that loosing a box is a rarit
Maybe this really depends on region, but I can't say this is true. I've experienced tons of hardware failures that have caused instances to throw weird errors, instances to randomly stop, instances to randomly disappear (along with their corresponding EBS volume).
Standard EBS volumes are only 99.8% durable. If you run a lot of instances on EBS, some of them will disappear.
GCP meanwhile has balanced zonal PDs (not even regional, zonal) at >99.999% durability.
I've never lost a disk on GCP, I've never even had an instance stop once without me telling it to stop.
Azure is absolutely a hot mess though. Had a VM a client was paying for backups on. Couldn't restore the backup because they changed generations of VM platforms too many times, literally no way to restore it. Had to get their support to eventually give me a .vhd that magically appeared in a OneDrive share a few days after asking about it.
But that's just a perfectly fitting excuse for GitHub degradation.
Web search, steaming, high frequency trading, and many other systems are resource demanding, and keep scaling. Whether to handle the surge due to bots or wider adoption. But GitHub can't scale git? It isn't even git failing.
Incompetence in leadership is what makes a tech business technically unable to meet growing demand.
Which wasn't a problem before MS acquisition. Git is the write intensive process. It scaled to millions of contributors. Then couldn't keep scaling. But sure if you believe the narrative and blame ai bots.
GitHub had 30 million users in 2018 when it was acquired by Microsoft. This year alone they added 30 million users for total of 180 million. So pre-Microsoft GitHub faced very different problems compared to post-Microsoft GitHub not even considering poor management or AI. They're by far the largest platform of its kind, so I'm sure the problems are uniquely difficult to solve.
I have trouble believing it’s a uniquely difficult situation when no other tech company of the same or larger scale has those same problems.
Is there a single Google service that has ever had reliability this bad? Facebook? Instagram? TikTok? Apple? Those companies all run hugely scaled write-heavy platforms. Why is GitHub so uniquely unreliable?
Me and 1 billion of my best friends can upload 5GB 4K videos to iCloud Photos and live stream to Facebook all day long but GitHub who is owned by Microsoft the second largest cloud computing provider on the planet can’t handle 180 million users pushing code and running pipelines on VMs?
I won't pretend to be an expert in the various systems of running these services, but to me it seems that GitHub's system is especially vulnerable to what AI is doing.
I'll give you an example: I pointed Codex at one of my repositories asking it to implement some features. Very quickly, it ballooned a 3 minute CI workflow to 30 minutes per commit, and the number of commits it started making increased 10 fold. So we're looking at a 100x increase in usage just from myself.
You factor that into the fact that all public repo get access to this same compute, basically unlimited minutes, up to 20 parallel tasks and 6 hours per request. Now because of AI, every small little weekend throw away side project is running a full dev ops shop. And the pressure never goes down because the AI just start a cron job to run these systems every single day, whether anyone is even looking at these repos including the authors, who may themselves just be bots.
And the systems are sophisticated: parallel builds across windows, mac, multiple linux distributions, multiple architectures, embedded targets, web targets, full feature matrix, the works. The default MO of these agents is to do as much testing as possible without any regard for resource constraints, and GitHub offers them unlimited resources, so it's like an addict meeting a dealer.
So in a sense, maybe being attached to Microsoft is the problem. Or at least their unlimited wallet is -- Microsoft is the enabler in all of this.
I also can't pretend to be an expert on this. Conceptually, everything you're saying makes sense.
It still just sounds like a really vanilla "horizontally scaling VMs" problem, especially for yesterday's incident that was focused on pipelines.
If this is a "GitHub is giving away more capacity than it has" problem, that's easily solved by rate limiting and queuing.
This is GitHub's explanation:
> During the incident, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.
> The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.
That sounds a lot more like a major architectural flaw and a really obvious oversight.
> Incompetence in leadership is what makes a tech business technically unable to meet growing demand.
I don't think this is true. Take LLMs for example, you need GPUs to serve them and these are in short supply at the moment making it hard to meet demand. I don't think this is necessarily the fault of incompetent leadership.
One of the big motivating factors in the development of git was the desire for decentralized version control.
With GitHub, we threw that away. It's centralized on steroids. Now we're all depending on one platform that has to be massively scaled to deal with the massive load of serving almost every major software project on the planet. And when it is down, we all notice.
I see this repeated a lot and IMO it's simply not true.
The main decentralized advantage of Git is that you can continue to do VCS operations without network access or access to the remote host.
Most of what Github does is managing collaboration. The only thing we've "thrown away" by using Github is the email based workflows or directly pushing git branches to various hosts. But unless you're gonna have team members SSH into each other's machines you'd still need a central repo somewhere.
I agree, no horizontally-scalable system is infinitely scalable in any dimension. You will hit some limit - for example, you might be able to support arbitrary scale in some dimension, but there is a bound on the sustainable rate of change, or the second-order rate of change, and you hit that limit.
You seem to be suggesting though that hitting those limits came as a surprise, and/or there was no future architecture plan to address that when it happens, which I think is the real problem here. You would expect a team with the maturity of Github to understand what those limits are, and plan for them, based on predictable growth in demand.
Actions is one thing, that seems like a difficult, dynamic and bursty thing to host, even before the Vibe Cambrian Explosion. And _nothing_ at scale is easy... But Pages? The static sites? Down for so many hours? Oof.
Use of Claude for coding increases the number of commits, merges and pushes. Add in difficulty in getting RAM, Disk Storage and Servers. A dash of high energy prices, data center angst and tariffs. A pinch of diverting resources to all things AI. You get legacy systems that cannot keep up with demand for expansion.
This is an excellent comment and I don't want to diminish by adding jokes but this totally reminds me of the hitler uses kubernetes meme. You can do everything right for HA and there'll still be something to trip you up.
AI slop-code has exacerbated the problem and GitHub is not able to keep up. If you look at the last 12 month's outage reports from GitHub's own status page, many of them mention capacity issues as an underlying cause.
As someone who has spent many years working in high load environments, this is not an uncommon pattern.
You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.
And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.
This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.
It could be that the extra usage just pushed them past one of those capacity thresholds.