I'd argue iPhone got ahead of the game then just stopped getting at all better. Siri AI is straight garbage, they won't let the interface be anything but what they want it to be, and the app store lock in/lack of ad blockers on browsers has made it basically just a phone for me. I have no intention of upgrading beyond my 12, and when it finally dies, I'm probably moving back to some kind of android phone if there's not a viable open source model out there by then (viable for everything, banking, etc). I only moved to apple because the watch was leagues better at the time than samsung's offering, but now I can't even claim that (and quit wearing it years ago)
Funny, I tried beta blockers before xanax for flying. They didn't seem to do anything, and the xanax I take a dozen times a year works wonders. Everyone's different.
I take Xanax when I fly. I have a prescription for a whopping 10 pills, and it usually lasts me three months. It does exactly what it's supposed to. "Banned, Period" is assuming your experience justifies taking everyone else's away, and those of us using them responsibly benefit quite a bit. 4x a day is also over recommended dosages, depending on strength - so that's more on the doctor than the drug.
Which makes some sense, I suppose. It was pretty expensive to use, which made it uncommon. These days, GPUs are so power hungry that'd you need a MASSIVE power supply to do it, unless you're SLI'ing low-end GPUs, which is kind of a waste. You're better off with a mid-grade GPU.
From a marketing perspective, yes, but technically there was a deeper issue with SLI (that intersects with the marketing issue): the GPUs don't have a lot of bandwidth between them so you have to be able to split work evenly across the cards without needing to copy work between them a lot.
3dfx did it spatially - each card rendered half the scanlines and then interleaved them - hence Scan Line Interleave (SLI). The problem is that this assumes all the work is going into pushing polygons onto the framebuffer. This assumption broke fairly shortly after 3dfx started doing this and it's an assumption that only makes sense for 3dfx's obsession with fill rate.
Nvidia wound up doing basically everything temporally: one GPU renders one frame, the next GPU renders the next. The reason for this is that "modern" (as in, 2004-era, so OpenGL 2.x / DX9) games do a lot of intermediate work that has to be available during final raster on both cards. They might render a bunch of depth buffers from various light positions to construct shadow maps. Or they might accumulate all raster operations into a bunch of color framebuffers and then compute lighting with the information in those buffers. Or you might just have a post-process gaussian blur that requires mixing pixels from across both GPUs.
Theoretically, none of this is a problem - you just send the pixels the other GPU needs. In practice even the fastest Nvidia SLI bridge only ran at something like 3.25GB/s and that's not even enough bandwidth to deliver a final 4K 120FPS image, much less transfer every intermediate result from every rendering step to both GPUs. The math gets worse when you start adding more GPUs. It's far easier to send a whole frame to each card, but this adds latency and stuttering[0] because you have to run the game two, three, or even FOUR frames ahead instead of just one.
That being said, there was a way to deliver multi-card rendering on Nvidia hardware, it's just that Nvidia had no interest in shipping Nvlink support in a consumer card because they wanted AI customers to pay more for it. So instead they washed their hands of it and made multi-card rendering an explicit API thing that nobody implements because <1% of people are going to spend $10k on a rig for slightly better framerates. AFAIK if you DO use multi-card rendering this way, you're doing all the synchronization across the main PCIe bus (which is actually more bandwidth than SLI ever had).
[0] Which, funnily enough, is the same problem with DLSS Framegen. Nvidia really, really likes building things that make the FPS number go up without actually delivering a better experience.
> [0] Which, funnily enough, is the same problem with DLSS Framegen. Nvidia really, really likes building things that make the FPS number go up without actually delivering a better experience.
I'd dispute that last part. At the very least, it depends on the game.
Cyberpunk 2077 with 4x frame gen is wonderful. With all the detail settings cranked to the max, it's a gorgeous game, but at that level, it even brings a 5090 to its knees. In 4K, without MFG, I often only get ~40 fps. With 4x frame gen, I get around 140 fps which is a significant improvement to the experience, and I don't even notice the artifacts unless I'm looking REALLY hard for them.
On the other hand, MS Flight Sim 2024 with 4x frame gen looks great unless I enable labels on landmarks and other aircraft. During camera angle changes, they get very noticeably distorted. The cockpit looks fine, though.
It's definitely one of the saddest stories in our industry. Peter Norton's books literally taught me to program as a teenager, and now his name is synonymous with mal/bloatware.
It is sad to again and again be reminded that even the ones we see as heroes earlier in our lives, could do something to kind of "tarnish" our idea of them. I think lately I've gotten better at handling coming across this, as I'm getting better at realizing we're all just humans, and what we see from these public people is often just a small selected slice, we're all flawed in our ways, but we typically don't get to see those unless it's something extreme, one way or another.
Many cases of this happening too, people loved and celebrated for years/decades, then does something that just absolutely tarnishes everything they would have been remembered for. Philippe Pétain comes to mind as an extreme example, one of France's greatest national heroes up until late in his life, he became head of the Vichy regime and collaborated closely with Nazi Germany.
Comparing Norton selling their company (even to a scummy owner like Symantec) to Pétain collaborating with Nazi Germany feels a bit ... extreme ... to me.
Not sure how it would affect home mortgages but the majority of icelanders are on mortgage plans that the principal remaining rises with inflation(I forget the name, I left a few years ago) - and will never likely pay them off.
Because they don't license it for and support it running on millions of different combinations of models, hardware, etc. MacOS can be clean only because it has such limited scope of what it has to handle compared to windows/linux
I live in a rural area and have only seen a Waymo a few times when travelling for work. Does Waymo require the user to have a driver's license? If they did, and get agreement of responsibility, that's one thing, but generally if not wouldn't it be Waymo's responsibility for bad programming or broken devices?
They don’t need a license and you are not supposed to touch the controls, so I don’t really see how you would be responsible. Waymos do seem to break traffic laws every once in a while, and I always wonder if mine will get pulled over and what will happen there.
You should be in your project description stating something like I know you are not a lawyer but use your best effort to help me find supporting legal information as by law I am allowed to represent myself.
I did this for a demand letter for a friend who was fired after reporting a sexual harassment claim in California - which is a legitimate duty to investigate.
I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.
You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
1. People add retries w/o backoff. Now we have a retry storm.
2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure.
3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns.
4. People find ways to fetch from multiple data sources to make a composite object but don't cache/reuse data they fetched, so one data source being down causes DOS on _other_ data sources because of retries.
5. People add failover mechanisms where all failures happily pick the same failover target because, ya know, it has the most free capacity.
6. People underestimate connection setup cost so "failover" causes huge increase in overall load due to connection setup (often tls setup) causing huge influx of "new" cpu work in a loaded system. CPU spike causes unhealthy destinations, causing more failovers, causing metastable failure.
7. People realize purely local decisions aren't optimal so add a layer of global health metrics (e.g. global retry budgets) but these systems add a time delay to the metric (ie its the view of the world 5ms ago). This delay makes their naive control algorithm go into oscillations or divergence spraying stuff everywhere.
All of these things have a similar pattern that when things are almost all good, they will make the system more robust. You'll get an extra 9 of reliability on good days. But they make everything far worse when things are bad.
Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
For resolution of a problem you can’t prevent categorically, you have to spend a good bit of time pretending to be the computer before you can trust the computer to do it. A run book is a draft of a requirements doc for a program to do the same thing. And you can’t afford to test that program in production without loads of simulations first. And that’s running the process manually and checking aggressively for any signs of problems.
Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO.
They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went.
A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else).
A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before.
Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
I think the "happy path" might be a slightly wrong classification in GP, since the post is in reply to a retry-storm issue and explicitly talks about retry storms and thundering herds.
I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse.
The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated).
The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification.
I've seen cases which had:
1. Retries on the client side (browser or mobile app).
2. Retries on the BFF.
3. Retries on Microservice A used by the BFF.
4. Retries on Microservice B used by Microservice A.
5. Retries on Critical Service C used by Microservice B.
Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C.
We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day.
A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination.
> I've seen many cases where engineers optimize the sad path, but pessimize the wretched path.
As said GP, yes this is exactly what I meant and "happy-path" was an unclear choice of words. They optimize for the unhappy path on the good days, and make the bad days much worse.
It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.
reply