chatgpt and grok linked to the the soundness issue, claude I am in usage timeout, and gemini shit the bed as usual: "The compiler rejects the program with error[E0597] on the assignment q.0 = Some(&l)"
And soon Firefox will include it in Stable. During October it will go from Safari only to majority coverage. Eventful month!
While AVIF may have a slight edge in some cases like in fairly lossy compression, you won't really go _wrong_ with JPEG XL (unless strongly CPU constrained) and I think the strength of the format is its extreme versatility as a "be all end all image format" for some time where comparative performance depending on functionality ranges from respectable to excellent. It can work as a replacement for AVIF, PNG, JPEG, WebP, even certain TIFF required scenarios (high bit depth, multichannel, layers) depending on use cases.
No, the reverse. Mozilla opposed Google's original attempt to push JPEG XL years ago, and declined to ship without an implementation that they could confidently say was safe. See https://hacks.mozilla.org/2026/08/intent-to-ship-jpeg-xl/ :
"We added experimental support for JPEG XL behind a flag back in 2021. But, at 100,000 lines of multithreaded C++, we were concerned about the attack surface this added to Firefox. So, we laid down a challenge to the JPEG XL team at Google Research: Build a safe, performant, compact, and compatible JPEG XL decoder in Rust, and we’ll ship it. That challenge was met; Google Research built jxl-rs, and it’s the core of our JPEG XL support in Firefox."
Years ago, both Chrome and Firefox had experimental support. In October 2022, Chrome suddenly announced their decision to drop it (citing dubious reasons like lack of interest or improvements over existing formats). [1] It was only later, in January 2023, that Mozilla announced their formal position of not really caring about the format. [2]
In October 2023, one of the JPEG XL devs said that the responsible team at Google was ready to write a decoder in Rust if that was the only reason holding adoption back, even offering to work on browser integration. [3] One month later, another dev confirmed that a different pre-existing decoder written in Rust was already standard-conformant. [4]
It wasn’t until September 2024, nearly a whole year later, that Mozilla changed their official position from not caring to waiting for a suitable decoder written in Rust. [5] And it was only then that jxl-rs development really picked up. [6] Only later on, Chrome decided to change their position as well, but we can only speculate on what actually changed their mind.
There was no Google shoving JPEG XL down everyone’s throat like they once did with WebP. There was no Mozilla taking a clear, principled stance from the get-go. Whether or not JPEG XL was worth it, neither of these companies has acted in a transparently fair, exemplary way. And of course the first major browser to ship JPEG XL support to users was Safari, which may well have played a role in the others’ decision-making.
Firefox did not take very long to support AVIF (another recent image format clearly favoured by Google). Please do correct me if I’m wrong, but I’m not aware of Mozilla expressing any reluctance over AVIF’s abysmal lossless performance or other shortcomings (some of which have been addressed since), or insisting on using a memory-safe decoder like they have for JPEG XL. Why the stark difference in treatment?
We may not be able to say for certain, but Google’s massive influence is by far the most plausible explanation I can think of. And if the first step in your decision making is looking at what Google is doing, then ‘following marching orders’ is suddenly not such a bad description. (One can argue that with 3 % market share they don’t have much choice, but that’s beside the point.)
Mozilla was a founding member of the Alliance for Open Media, having sponsored Xiph to create Daala, which was later polymerized with Google's VP10 to create AV1. Mozilla has a vested interested in pushing all things related to AV1, including AVIF. That has nothing to do with Google's favor and everything to do with Mozilla's favor. Their hope was to use the existence of AVIF as leverage to give hardware vendors more incentive to provide hardware support for AV1.
> Mozilla has a vested interested in pushing all things related to AV1, including AVIF.
Why? AV1 was developed as a video codec. That doesn’t imply it’s as good of an image format nor that everyone who supported its development has vested interest in making it into one. Just because someone has made a cool hammer doesn’t imply they now need to view everything as nails. (This provenance even brings some drawbacks with it, yet this time seemingly no one took a pause before tossing in yet another image format.)
> Their hope was to use the existence of AVIF as leverage to give hardware vendors more incentive to provide hardware support for AV1.
This is the first time I’m reading about this and I’m having a hard time seeing how decoding images would make for a significant incentive to provide hardware support (especially when everyone was already more or less on board with AV1 as a video codec). But I could be wrong, of course. If you can provide any links where Mozilla gives this reason for adopting AVIF or any hardware vendor gives significant consideration to decoding AVIF in hardware, it would be much appreciated.
[2] was anticipatory obedience from Mozilla. Good for them that they grew a spine later (or more likely: they knew that Google will implement it some time later, so the change of opinion cost them nothing).
This is mental gymnastics. If Mozilla were following Google's orders we would not need to resort to phrases like "anticipatory obedience" to describe a neutral non-position; if they were following Google's orders, they would have simply endorsed it, end of story, and then they would have revoked that endorsement in lockstep with Google as well, which they didn't.
Again, when exactly has Google endorsed JPEG XL? When did they promise to ship it in Chrome stable? Sure, they had it in development as an experimental feature at first, but so did Firefox.
The way I read it, both companies were tentatively interested until Google said they changed their mind, and shortly after Mozilla decided they didn’t really care. Correlation doesn’t imply causality, but their positions followed a somewhat similar trajectory at the very least. (In contrast with, say, Apple.)
I just don’t see where you keep getting the idea that Google was at some point far more supportive of JPEG XL adoption than Mozilla was.
Rollout looks solid, but I'll probably avoid deploying it to anything live til early next year at the minimum. That'll give time for browser adoption to happen.
Biggest concern I have was the same I had with AVIF a couple years ago, which is hardware acceleration on the server side. I attempted to roll out AVIF only to find out the encoding time was so long that sync calls were timing out due to queuing. It's gotten a lot better, but just know that all of these newer formats really do need more compute. It may not be worth the cost tradeoff vs webp until encoders are baked into chips.
I had the same problem with AVIF. My platform allows users to “scrub” through a video file they’ve uploaded by hovering over the thumbnail. To achieve this it creates a grid of 100 representative frames, up to 480px in width per frame. So the images are pretty large and take up a lot of space in JPEG format. File size is important because it reduces lag time before the thumbnail is scrubbable.
We tried AVIF and file sizes were much better but it would take multiple minutes just to generate. We also got feedback from our self-hosted customers that it was using way too many of their system resources.
So we’re now using WebP which generates in about 10 secs using much less CPU. I’m open to JPEG XL if it can improve on those metrics.
How are you generating the frames? Is there a chance you’re looking trough the frames at regular intervals and have to actually “render” the video (I don’t know the proper term for this)? Did you try just extracting the keyframes where an actual full image is stored in the video file instead?
> your concern specifically applies to images that your application server is rendering custom for each visitor?
It's more if you accept image uploads from users in any way. I experienced a ~10-20x increase in processing time when accepting AVIF images, and I found that I couldn't rely on sync functions for that. I had to both change to async and upgrade hardware in order to support it at scale. I run a small reddit-like site that accepts media uploads and the end result was I had to choose between unshipping AVIF or doubling my server costs. I found that the image savings over webp weren't worth it at the time.
The same story is likely true for JXL until there's hardware encoding.
Thanks! I didn't think of the decoding story but it hints at a possible reason why some sites that have a high "usage to money" ratio might specifically seek to avoid the new formats if they need to process the images, which I suspect most do, even if it is just to create a thumbnail.
* I don't deploy anything not supported in current Firefox ESR. That's June-July 2027.
* The versatility is a bit of a downside when it comes to lossless. Programs play games when it comes to formats mixing both and when may appear to be lossless isn't always thus. It would have been nicer if there was a direct way (via extension+mime) to indicate lossless is expected.
Quantized PNGs are a thing and those are not easily detected either. For JXL, most files compressed with Modular mode are lossless. You can detected them in the metadata
The problem is not that a file with a lossless extension (e.g. PNG) may have actually had lossy processing.
The problem is that it's not necessarily trivial to tell programs/pipelines/transforms to act only losslessly (there's definitely no common switch; Quality 100 is in fact still lossy in some encoders/formats). In practice some programs use the extension to realize what they should be doing.
This doesn't really solve the problem, but the good news is jxl has excellent generation loss in lossy mode compared to other formats, if you recompress it with high quality, it keeps original quality pretty well.
The article brings attention to advancements in AVIF encoders over the past few years (to which its author has greatly contributed), but does not provide a definitive comparison between the two formats, even when it comes to the web. Parts of it are entirely based on personal opinions and educated guesses.
There are some unanswered questions about the methodology used, the resources spent on development of the respective encoders is not taken into consideration, the future developments of JPEG XL encoding are written off as too costly and the author’s opinion on what kind of content is appropriate for the web is quite subjective. See also the recent discussion:
Again, JPEG XL is not currently optimised for low bpp images. Compared to AVIF which just spend the past 3 years optimising it before writing that blog post.
JPEG XL has been designed for keeping the image intact at the lowest possible bits per pixel (BPP), usually around 1 to 1.6 BPP.
AV1 was designed to deliver a pretty good new frame with as low as possible budget, and further video frames could bring the missing detail later. AVIF/AV1 has much more aggressive filtering and more features that are well suited for low bit budgets -- but also is missing features that are needed to get things right at higher quality.
JPEG XL can control filter strenght at 8x8 pixel resolution, whereas in AVIF filter strenght is constant over a 64x64 block.
Similarly, adaptive quantization is more fine grained in JPEG XL.
Finally, adaptive quantization automatically modulates the filter strength in JPEG XL (but not in AVIF), so the same data does not need to be sent twice for this purpose.
All of these three differences are visible in bit budgets only in the higher quality images.
that article does not seem specify which bit rates/quality targets were used for analysis, and uses a 1980s scanned photography film scan corpus for the claims ... let's use digital photography test corpora nowadays!
I'm sad that it doesn't work particularly well for low quality thumbnails. AVIF seems better in that department. You did say that but it means I can't just use it for everything.
I don't see any advantage of any of the new formats over the good old and proven formats, so I stick with PNG and JPEG (properly optimized.) I can fire up any old browser and it will just work.
Unless you're targeting really old browsers, lossless WebP is absolutely worth it over PNG.
As a test, I have a set of 36 old MS paint Drawings, originally drawn on Windows 98SE, originally saved in bitmap format with a total size of 53.4 MB.
Transcoded to PNG format using FFmpeg and the highest -compression_level of 100, this same set of images takes up 403.8 kB.
Transcoded to -lossless WebP format using FFmpeg and libwebp with the maximum -quality value of 100 and maximum -compression_level of 6, the set of images takes up 174.8 kB.
Transcoded to lossless JPEG XL format using cjxl in the highest compression settings, the 36 images only takes up 126.6 kB.
Given the computing power required to decode JPEG XL images and their limited support, it may not make sense to use JPEG XL for non-archival purposes, but lossless WebP is an excellent middle-ground, achieving over 2x the compression level of PNG.
Lossless WebP has also been supported by all major web browsers since September 2022, when Apple finally added full support for lossless and animated WebP images. It really doesn't make sense to still be using PNG, unless you're targeting really out-of-date Mac or iOS devices. Having drawings and illustrations be 50% smaller (versus PNG) is huge!
Personally, I avoid WebP out of spite. While it has greatly improved over previous formats in certain uses, it has been lacking or offering only marginal improvements in others. That didn’t stop Google from pushing it hard on the web.
I was left feeling like the next time we adopt a format that tries to do everything, it should actually do everything just as well or better than its predecessors. Lo and behold, JPEG XL is probably the first format which actually achieves this, covering all kinds of pictures you can think of and doing a pretty good job on all of them. So for the first time, I feel like the format has actually earned its place in the ecosystem, rather than just being some megacorporation’s personal favourite.
Of course, this point of view is emotional more than rational. Technically, WebP is a sensible choice, so I don’t mean to discourage anyone from using it today.
It wasn't some one company's choice and push. Almost everyone wins when bandwidth is cut and you get higher quality for less space. I'd argue most people's hatred for webp purely comes down to a few other corporations like microsoft, not supporting it well in the early days and blaming the format instead.
How so? This table [1] shows the adoption dates for WebP in various popular programs (as does Wikipedia [2]). Of all high-profile programs, Chrome was the only early adopter; other browsers were avoiding it for years, not to mention off-line software. Mozilla went as far as to work on a JPEG encoder [3] after WebP was introduced by Google, to see if maybe JPEG itself could be improved enough. How does that not read like one company’s choice and push?
> Almost everyone wins when bandwidth is cut and you get higher quality for less space.
Sure, but that doesn’t mean you immediately adopt every new codec that brings improvements in some area. With every new feature comes new code, new baggage, new potential attack surface. You have to ask if the improvements are big and consistent enough to go through the effort and burden the entire ecosystem with yet another format everyone now has to deal with. You need to think of the user and developer experience both in and outside of the web.
Microsoft would have been perfectly within their right to blame the format, since Google just decided to let it run wild on the web without garnering more support for it first. The format has its pros and cons, and it’s up to all the involved parties to weigh them and hopefully reach a consensus.
I'm sure they have some preferences for smaller pages.
If you're saying they did a thing where 200KB webp had an advantage over 200KB jpeg, please link your source. When I search I only see people saying that's not the case.
JPEGXL literally is able to display images and colors that jpeg and png cannot (32-but floating point, layers, emission independent transparency, spot colors, built im depth maps). It’s not just about compression optimization. So for those use cases standard JPEG and PNG alone will never work. There are “updated” versions of JPEG that support some of these like HDR but those won’t work with any old browser either so back to the same issue.
Bottom line it can deliver greater quality at much smaller filesizes. Considering RAM and storage are more expensive per byte now than they have been in years, I think it's incumbent on everyone in a position to do so, to use those resources wisely.
WebP and AVIF and HEIC had similar advantages but also some disadvantages (honestly chief disadvantage to WebP in my humble opinion was that nothing except web browsers -- to this day -- seem to support it!).
Another important point in its favor is that JPEG XL is designed as a royalty‑free, open standard, and Google provides a perpetual, worldwide, royalty‑free patent license for its reference implementation.
Avoiding JXL (once the rollout is sufficiently complete) seems, to me, like still shipping (non-animated) .GIF files where .PNG is better and smaller.
I imagine the result of this will be social media platforms and such will deliver higher quality images at the same filesize. Currently for local storage any image format is mostly fine, but for web services the image size becomes a massive cost which is why they compress your 8MB JPEG down to a 2 megapixel crunchy mess.
It's trivial to make an image that you can see the 8-bit quantization in. Just make 256 vertical gray bars, equally spaced. In much of the image (usually the darker parts, but depends somewhat on monitor calibration) you will see the border between two values. Note that this is every single possible grey value in 8-bit RGB, so it is a proper un-dithered rendering of a gradient, and the borders you are seeing are definitely quantization artifacts.
Now, if you apply proper dithering, particularly on a high-dpi screen, you can make it look a lot better, possibly even undetectable.
And this is just with the dynamic range that is comfortable on a computer monitor! If we really want to represent vibrant images, then the glare of the sun off of shiny objects should be much brighter than the "paper white" background used in e.g. your word processor. It should be intuitive that the brightest highlight in a real-world image ought to be too uncomfortable to fill your entire monitor with...
Lastly, there's a question what and how the image gets to the browser. Sure for the images on the landing page for $LARGE_CORPORATION you have a graphic designer make everything look great, and since most people still have 8-bit displays, you'll want something that looks good on one. But what about e.g. a site for uploading photos? Are you going to quantize all of their photos before displaying them (possibly to someone with an HDR display)?
The cost of HDR displays is decreasing, and the data cost is minimal. (adding more bits to video actually decreases the size because the rounding errors get smaller)
Yes, a hdr photo in heif or jxl is a night and day difference. And if you apply any kind of color edits to the photo it stretches the colors to the point visible banding occurs on jpeg.
Yes, and image editors absolutely can. If JXL lets me correct over/underexposed images by default everywhere I find them then it's worth it even just for that reason alone
E-commerce is the important use case, particularly in goods where people care about appearance: clothing, cosmetics, food.
Things in these categories have colorful packaging with spot colors, metallic inks, coatings, as well as store displays with carefully designed lighting. The added cost is justified because it increases sales. It makes sense to carry that over online.
JPEG can represent "X" number of colours, but the human eye can detect Y>X colours. JPEG-XL can represent >X colours (but not as many as they human eye, Y):
JXL can do more colors than the human eye can perceive. Some chunks are out of range when using current standards, but it can do other chunks at 100000x the color resolution your eye can detect.
It looks fine sure, but that doesn't make it good either. Are you going to lock display technology to 8 bit forever? Most PC monitors already match or exceeding that limit, phones especially. HDR is something that needs to happen eventually, and for that new format is very important
I used to think this and then I saw how well webp reduced my website's bandwidth costs compared to optimized jpg or png. I felt a bit left behind at that moment, like I should have been paying attention to these great improvements over the years.
It is great to reduce file size but also the quality drops faster than it does with JPG. I did some testing and could confirm, it is quite documented.
Which I think it might be one of the reasons why Instagram still uses JPG for their images. The rule of thumb is that if your product is picture quality sensitive (like photos in IG) then use JPG, if not then WebP is fine since the compression is higher.
Are you actually comparing apples to oranges? If webp reduced your bandwidth that much you're properly overcompressing way below the quality you used for jpg.
I'm not sure if the parent comment means to say, but I could choose to interpret their meaning to be:
PNG/JPEG have strong support (nearly everywhere). I think of them almost as the modern TIFF - relatively simple, and almost everything supports one or both of them (few to no issues).
JXL may eventually get there (that indeed seems to be its goal), but for now I wouldn't drop e.g. PNG or JPEG support in favor of JXL now.
And arguably, perhaps never drop support; its such a prevalent format, even if all software (prevalent or otherwise) supports the format in 10 years, there will be such a huge stockpile of JPEG/PNG that something will need to be used to read/convert.
Not everyone needs to be an early adopter of every technology.
Even if JXL is superior in every single way, it doesn't make sense to throw out a perfectly functional graphics package and design language for a marginal bump in capability.
The industry already started throwing out jpeg for heif because jpeg is so woefully inadequate. Most camera brands have heif support now despite heif having horrible patent issues. Jpeg XL provides all the technical benefits without being patent encumbered.
For a past customer, not at all. That customer really cared about delivering image-heavy pages quickly. There was a fairly big difference in customer perception at a timing threshold.
The very point of this topic is that JXL is leaving the "early adopter" realm. If your website begins serving JXL assets next month, you'll simply be an adopter.
I just switched over some machine vision code to use it for the lossless format, saving around 30% file size (~5 gigabytes per dataset) over optimized lossless PNG. I'll gladly take any storage donations you may be passing out though, so I can continue to use the good old and proven formats! ;)
biggest advantage for me is file size and lossless conversion of existing jpg images. In a network scenario that can meaningfully impact load times. Others will care about e.g. bit depth
Yeah but file size and lossless conversion of existing JPEG are something done by many other codecs. Dropbox famously developed Lepton[0] to compress existing JPEGs losslessly. There need to be some other factors that would lead to increased mindshare.
Being a well supported format on all OSs will allow that. One of the biggest failure of WebP was very poor native support beyond browsers. JXL is supported natively by pretty much all operating systems right now beside Android. JPEG recompression is simply an advantage
I think the main issue with webp is it was pushed as purely a web delivery optimisation format and not a final output file you’d use in software. So it worked fine if you have a website automatically converting files for delivery, but no one was pushing for it to be supported in other programs.
JPEG XL is marketed as a high end format for photographers and already has support in most editing software.
There are many websites that would happily deliver webp images to the user and then refuse to accept webp images uploaded by the user (in forums for example). It just didn’t work for anything other than being displayed in a browser.
well, lepton doesn't have browser support. I'm not married to jxl specifically, I just want something in the browser with those features, and jxl fills that niche
I still miss the point. We have connections nowadays that are mostly good (with even the average domestic connection being gigabit!) and JPEG was fine in the days we had much slower connections.
On the other side we introduce yet another format that has to be supported, that creates compatibility issues with people that is using older browsers, or older software, older cameras, etc.
It's just another format meant to replace other formats to uniform all, so now we have N+1 formats that the user has to deal with.
What are the benefit for the final user? A reduction in 50% of the image size? We are not talking about videos (I re-encoded a lot of older videos to h265 because it save me almost 100/150% of the space, but unless you are a professional photographer, that btw uses raw format for archival, you don't have Tb of images so the saving for the average user are neglectable!).
This strongly depends on where your domicile is, even in Europe and the US. And lot of the users live outside these areas.
> JPEG was fine in the days we had much slower connections
Except that waiting for a large picture to load was very much thing, and in many cases still is.
> What are the benefit for the final user?
A number of users are still on metered mobile plans, and have little storage on their phones. With the current situation in electronics production, new phones are not going to offer more space, cheaper; to the contrary.
> This strongly depends on where your domicile is, even in Europe and the US. And lot of the users live outside these areas.
The rest of the world has faster Internet if anything. In general, when new Internet backbones are set up they will use the fastest tech available but then only slowly upgrade it so paradoxically regions that are lagging behind in tech can often have better Internet connections.
They also spread it thin. I see how places like Goa have very cheap and very fast 5G internet. Places like Botswana, not so much: the speed may be there, but the cost is very different. Mobile internet in India may be very inexpensive if you come from London, but it's still far from a negligible cost for locals.
The issue for me is that RAW sucks, and Jpeg also sucks. Jpegs are standardized and display anywhere, but are also super lossy. RAW, like fish, is not actually a thing beyond a sorta "I know it when I see it" deal. It's non-standard even between camera models from the same manufacturer, and there are like 10 pieces of software that bother with this bullshit and afaik none of them are OSes or browsers.
There is actually a standardised raw format, .DNG. The newer camera brands use this but the big players all stick to their proprietary formats. Amusingly DNG internally uses JXL to store the image.
First, try explaining to a non-tech person the difference between jpeg and png and they commonly scratch their head. It's nice to have a single format covering lossy and lossless, just meaning "picture". Whether it's a photo where lossless is ok or some icon where it isn't should be insubstantial.
Second, the "all connections are fast, what's the point" is one reason why the web is so slow. Connections are often slower than the dev with his 1gbps connect feels every day. The average website loads slower (in wall clock time) than it did 15 years ago. Smaller files and fewer network round trips would do everybody a favor.
I checked price per AA task and yeah Argon vs Opus 5.5 High are practically neck and neck both on "intelligence" and cost per task. So if you aren't pushing higher on Opus (which I think can quickly become very expensive and I always treat Max and those like "benchmark settings"), I think this becomes more of a matter of which platform you like more or prefer for various reasons.
Honestly, I think this will be a trend in 2027 when all those models become "good enough" for elite coding and whatever. I predict they'll have to branch out more and build their platform to differentiate themselves from each other, maybe even in terms of branding and trust, marketing towards various demographies, youth vs elderly, students vs employees, etc.
They test it on launch day, then benchmark it against that. A deviation of above 10% is considered a change. They're currently tracking Opus 5.5 and GPT-6 Astra.
This bench famously detected a degradation of Opus 4.6 which Anthropic later blogged about. I personally think people sense nerfs more often than they happen and that it's often about honeymoon effects.
Used to work on a chat app where we had full control of the stack from the GPUs to the chat interface and everything in between. We were a small team too so I could be pretty confident that nothing changed in the stack and we still regularly had users complain that this or that model got nerfed. Perceived performance is actual performance over expectations and the latter just keeps increasing over time.
It doesn’t mean the big labs don’t also nerf models! But if they didn’t you’d still have users complaining.
For years I used to buy ASICS running shoes. Every year they released a new model of each shoe: "Nimbus 23", then "Nimbus 24" the next year, etc. And every year people would complain in the user reviews about how each shoe was worse than the last.
I was like, wow, I guess the shoes must be literal torture devices full of MRSA-covered broken glass at this point. They've been getting continuously worse for 24 consecutive years!
Of course, what was really happening is that they were not getting worse, but naturally every year there was some small percentage of vocal dissatisfied users, while the silent majority simply enjoyed their shoes and didn't have much to say about them.
(The sorta-opposite happens in sneaker reviews as well. People will gush about how cushy the sole in some particular new sneaker is. Well, yeah, of course it's cushy -- you're comparing a new sneaker to your old sneaker where the foam had lost its bounce...)
I wonder how much of that is caused by folks actually noticing actual degradation in product quality over time (whether or not this exact product is suffering from it).
Shrinkflation is a thing, which people suddenly started noticing in the past 5 years.
There's also "the Schlitz Mistake", which I've heard summarized as "most customers won't notice if you take your product's quality from A to B (or C), but they definitely will if you take it from A to J (or A to M)"
At this point I kinda assume that any company releasing year updates to a physical product that _doesn't_ take the opportunity to trim costs / reduce quality would be vulnerable to a shareholder lawsuit for leaving money on the table...
> I wonder how much of that is caused by folks actually noticing actual degradation in product quality over time (whether or not this exact product is suffering from it).
That's certainly common!
I think ASICS' running shoes were a fun example of where this was probably not the case.
- I certainly didn't notice a difference in that time, though I'm admittedly not much of an actual runner
- Serious runners might notice small technical differences, but the negative user reviews didn't seem to indicate those were the people making the complaints
- The overall user reviews remained positive
- The competition in the shoe market is incredibly fierce; I'm not sure a brand could tank their quality and survive for long
- Let's not forget the other big variable: the wearers' bodies, particularly their feet. Now, those definitely do change over time -- most often for the worse, sadly!
- I doubt anybody was blind A/Bing a pair of Nimbus 20 against a pair of Nimbus 21 or 22 or 23 or 24. At best, a longtime Nimbus buyer is probably comparing a brand new pair of e.g. Nimbus 24 against their degraded but broken-in Nimbus 23 and their memory of how the Nimbus 23 felt when new. (And their body is a year or two older at that point..)
- Because it's such a long-running line of shoes, there could certainly be year-to-year variations... but it's hard to imagine there was an actual 5 or 10 or 25 year downward slope. I mean, otherwise at that point the shoes would just be instantly injuring you or falling apart in a week
- Also because it's such a long-running product line and (aside from bleeding-edge professional marathon/track shoes) sneaker manufacturing in general kind of seems like a solved problem... it seems like all of the possible cost optimizations have already been optimized. I don't really think there's much of a manufacturing or bill-of-materials cost difference between $20 running shoes and $200 running shoes anyway -- I'd be pretty surprised if "quality shrinkflation" was really much of a viable way for ASICS to save a few pennies.
I'm curious how common such lawsuits are. I don't think I've heard of any specific instances where shareholders sued because a company didn't make the product worse.
On the other hand that idea that "companies exist to make money for shareholders, to ONLY make money for shareholders, and doing any other than maximizing shareholder returns is bad" is pretty commonly accepted (and, I believe, enshrined in US law)
It's pretty typical that physical-goods manufacturing "optimizes the process" to cut costs during years 1 & 2 of manufacturing.
Ikea is notorious for this: The early Billy bookcase had heavier veneer and sturdier construction early on, and was actually a really good purchase for the money. The later years replaced veneer with paper foil, used thinner shelves, frames, and backing panels, and was just significantly weaker.
Amazon Basics is incredible in this regard, they’ve optimized SKU identification down to a pipeline. They’ll essentially randomly pick items off their internal list of highest netting sales and test them to see how dependent they are on brand name recognition and price-quality signalling. To do this as efficiently as possible, they simply purchase a few hundred units of a high quality product in the space, stick it in an Amazon Basics box and list it on their site under their Amazon Basixs brand at a price they feel they can achieve via white labeling, and wait to see how it sells. The use of high quality items (with quality above what can actually be had at the listed price point for the duration of the experiment) means they are really only testing the user base’s willingness to forgo a brand name for the category in exchange for a discount. If it sells well, they then work on sourcing it in bulk as a white labeled item “for real”, while if it sells poorly they simply delist and move on.
I (used to) buy pre-spliced/terminated fiber optic cables with some frequency from Amazon and came to be familiar with the brands and their quality. One time while shopping for some fiber optics, I saw Amazon Basics-labeled OM-3/OM-4 MMF cable at a very tempting price, so I purchased some to see if it was any good.
To my utter shock and surprise, when I received the trademark plain cardboard boxes with the Amazon Basics label on them and proceeded to open them, I found that I was sent boxes of cables still factory wrapped with labels that clearly read “Corning Optical” – which if you know anything about optical fiber, was pretty much the premium brand in the game. I should have stocked up because the next time I went to order I found out their experiment had ended and they no longer sold “Amazon Basics” finer cables.
Amazon Basics AA NiMH was well known to test exactly the same as the top Japanese brand 'Eneloop'. Extremely good specs all around
Recently though, they are still called Amazon Basics but no longer test like Eneloop. They've changed manufacturers for the worse and are hoping no one notices...
I did a deep dive into NiMH batteries a few years ago and concluded that most people felt similarly: you can often get "good" (similar to Eneloop) specs for a short run from almost any manufacturer at the outset (see Ikea Ladda batteries - suspected of relabeled Eneloop for a while, but now not as good), but consistent quality is pretty much only Eneloop or other name brand, with Eneloop generally being the best.
In the interests of saving my sanity and time (it's not free!) having to chase down which batch of which brand is "good" at the moment, I just decided on Eneloop all the time. Sure, we now have like 100-120 or something (wife likes flameless candles - just bought another 16-pack AAs) and I COULD maybe have saved $200 by buying dirt cheap. But all the time spent debugging flaky batteries, having the spouse complain, etc wasn't worth it to me (I get paid reasonably well).
Yeah, name brands like Energizers are consistent but slightly worse than Eneloop at roughly the same price.
I actually made a battery tester as a hobby electronics project. Fully dumps the energy while measuring mAh and seeing some measurements of internal resistance.
Something I did notice was that crappy battery chargers can permanently damage even Eneloops. So don't cheap out on the chargers either.
The high speed chargers (2 hours or less) are on the edge of what is safe and could permanently damage a cell. The 4 hours or slower chargers are just way more reliable. There's probably someone out there testing different chargers (trying to find the safe fast chargers) but given how cheap NiHMs are (even the "expensive" Eneloops), it's just easier to use 4 hour or 8 hour chargers and have masses and masses of extra NiHMs laying around.
Im sure you know this, but do note that fully dumping the charge can a) damage the battery and dramatically shorten its lifespan, b) give you different results depending on your discharge rate and the batteries you are testing, as they all have different current-dependent discharge curves.
Oh yeah. I stop at 0.95V. which I consider fully discharged (but not so low that I'm damaging the cells). I'm about 10% off the manufacturers claimed specs but I'd rather not push the limits of testing. That's low enough to differentiate between good and bad cells
The discharge rate is currently regulated by only a 2.2 Ohm resistor. The next version of this circuit will be a constant current drain circuit to make all tests draw the same mA across the whole test.
Right now more current is drawn at 1.35V start and far less current is drawn at 0.95V end of test.
--------
The part that they don't tell you is that 0.95V isn't one point. When you disconnect the cell, it charges back up (kinda like a capacitor). This process can take multiple minutes (!!!!). I've defined the end as being lower than 0.95V for more than 30 seconds, even if disconnected.
Except for that one year when they completely swapped the meaning of the Cumulus line, which I think was 2008 with the cumulus 9 to 10 transition. It went from a neutral shoe that was good for people with high arches to more of a stiff stability shoe. The complainers aren't _always_ crazy. :-) (That doesn't mean the cumulus 10 was worse, of course, it just was a more substantial change that affected the type of runner the shoe was designed for.)
Wow the old grumpiness that lingers in my head from losing my favorite shoe. Who knew? Now I'm old and heavier and run in the Nimbus and am happy again. But you're right, of course, that most of the model changes are just fine and people like to complain.
I find that I learn to "trust" a model to get certain things right, as I would trust a colleague. So, as `expectation` increases, my prompting and context management gets sloppier.
`percieved_performance = actual_perf/expectation`
`expectation` is an increasing function over time.
`actual_perf` is a stochastic function of the model's true ability, context, etc. -> a recipe for some bad sessions.
As for multiple bad sessions in a row, this is a studied phenomenon in gambling where players perceive "runs" because our brains love to find patterns.
This reminds me of the fact that true random does not feel random to users due to the clumpiness that the average person does not anticipate existing in true random.
e.g. The original apple shuffle and the Risk app ins which a string of songs from the same album or three one roles are "not random"
I've been working on a game that has dice rolling and even knowing about this effect, I started going crazy yesterday when I had a long streak of numbers, like 1-20, it was 15-16 like 9 times out of 10. I was sure there was some kind of bug in how it was initializing random, or saving the number, etc etc. Just could not find it. Streak continued to another roll, another roll... still couldn't find it. Then the streak just broke. Apparently just random being random.
> I've been working on a game that has dice rolling and even knowing about this effect, I started going crazy yesterday when I had a long streak of numbers, like 1-20, it was 15-16 like 9 times out of 10. I was sure there was some kind of bug in how it was initializing random, or saving the number, etc etc
Been through that this week as well with 100% success on 40% odds over multiple iterations on my game.. I tend to not dig into random but just rather ensure it works 'as closely to intended' as possible..
I find it useful to consider something like shaken rice. If you take a 10x10 grid and shake 100 grains of rice on it, you'll find that some cells contain no rice while others contain as many as 5 or 6. Run the experiment enough and you'll converge on each cell getting one grain per run, but any individual sample will likely be clumpy and the state in which each only has one will occur infrequently.
Also, consider that in flipping 10 coins, you'll find strings of 2 heads in ~86 percent of runs, 3 in ~51% of runs, 4 in ~25% of runs, 5 in ~11% of runs...and in strings of 100 flips you'll finds strings of 6 in ~55%, 7 in ~32%, 8 in ~17%, 9 in ~9%...
Widening the range from "rolling exactly 15" to "rolls 15 or 16" or "rolls between 14-17" makes the strings even more likely as you're doubling the success rate from "only 9 15s" to the "any string between 9 fifteens, through 16 and 8 fifteens, to 9 16s" space.
To check if your random is randoming you can calculate expectations versus your results (using a large enough sample) with:
For N samples of a fair die, expexted runs k with probability of success p and failure q can be calculated as:
General Variables:
N = total number of rolls/trials
k = target streak length
p = probability of getting the target outcome (e.g., 1/20 for a specific roll on d20 or 1/10 for two specific results)
q = probability of getting any other outcome (1 - p)
Expected runs of AT LEAST length k:
E(runs >= k) = p^k * (1 + (N - k) * q)
Personally, I find that 'sticky' dice always provide a nice narrative device, at least in narrative games. A character who's player can't seem to roll over a 10 must, after all, be cursed or perhaps deliberately sabotaging the party.
Took a little bit of digging to find it - I remembered the graphic at the top and found a blog post that copied it and linked to the blog post, but the blog post isn't there anymore... so web archive.
> I personally think people sense nerfs more often than they happen and that it's often about honeymoon effects.
I believe in temporary nerfs. Operators reducing quality significantly to increase throughput for whatever reason (high demand?). Same session, model being completly incapable, and it being back to normal the next day. Experienced that with Anthropic models way too many times. Never on weekends, usually during US work days.
It's been a few months since I last recall this though, must have been pre-opus-5. And I know there are benchmarks for this as well, hence "believe".
I vividly remember when ChatGPT3.5 went fully mainstream, there were times where within minutes you would realize they were only serving up idiot mode and there was no point trying to do much until demand died down and they swapped back to the non-quantized version.
People called it lazy mode, in that instead of writing the script you asked for it would basically tell you to learn to code then check out xyz topics to tackle the problem.
Regarding lazy mode, I recall ChatGPT sometimes almost refusing to do a web search despite me asking explicitly for it, instead replying with speculation about what the search results likely would tell us. If I pretend to be angry that it didn’t search the web it would however do it. Haven’t noticed this in a while either.
It's like doing the calculations yourself (with whatever tools you want) and knowing what you're doing, steering the process yourself, rather than begging someone to give you the result so you can then show it off like you did the work. I can understand and explain what my calculator is doing and I'm not offloading decisions to it. Using a language model to shit out projects that you can't fully understand the structure of yourself is insane. Ceding control of any design decisions to a language model is insane. They're fine as second-order autocomplete and semantic search engines. Humans should be handling design and implementation entirely, making things for other humans. An LLM running in a loop can't build humane systems.
> Humans should be handling design and implementation entirely, making things for other humans. An LLM running in a loop can't build humane systems.
I tell the LLM what to do in its loop. It builds it. I tweak it until it is perfect. I care a lot about UI.
I'm mostly building my own UIs lately, but I find no problem with the development loop. I'm building much better UIs, because it's way easier to test things, and scrap things that I thought would work, but don't. It all happens in a matter of minutes.
Anecdote - I work in a time zone offset from continental US. The performance of anthropic models would noticeably drop, around the time US work day started. It was so bad around 4.x time that multiple colleagues re-arranged their schedule to have least overlap with US work day. Admittedly it's been better recently.
Friend of mine works for a corp that is one of the top spenders on Claude models. He complained about these nerfs during peak demand.
Their Anthropic contact changed something and it did not happen since.
This is what pissed me off the most. Make it slower, rate limit it, move the credits to other time slots, idk.. but returning BAD results? That's the worst approach you could take.
I do remember times in the past where when I was up super early (4am EST) I would get super high quality results, then in mid afternoon EST it seemed to be degraded.
Anthropic A/Bs my weekly quota amount. So I have an automated prompt that runs at 3 AM with a transcription task, I measure input and output tokens, and weekly/5 hour quota before and after. The absolute token counts stay within 0.1% while in mode A it counts for 1% of my 5 hour quota and mode B 4% of my 5 hour quota.
How did pissing off your customers ever become a business model?
I can't imagine sticking with a supplier that plays games like that with me. Tokens are a pretty vague quantity to begin with (you don't control how many tokens a model puts out in response) and giving a couple of purposefully wrong responses will happily inflate your bill, but you don't care because eventually it worked. It's almost an ideal vehicle to scam people.
Imagine the power company being able to decide how much you consume and at which price point.
I find it amazingly rich that they bill you for """thinking""" tokens and now you don't even get to see them, they're gonna train the thing to sing "99 Bottles of Beer on the Wall" to itself before it starts work.
They could, but as you see here, people are very eager to create dashboards and trackers that do external accounting by proxy, so they can't just "make up numbers" without the customers noticing and making a fuss.
I think it's sinister, but not for the reasons you're thinking. I think they're just wildly unprofitable on subscriptions. The idea that most customers won't use their full quota is plain wrong: most people are maxing out their subs, or even reselling whatever quota they have left.
When you're running something at a loss, you can mistreat your customers and they'll still stick around (I'm an example). OpenAI and Anthropic are now cheaper than Chinese models on subscriptions, while being 6-10x more expensive on the API.
My guess is they need the user numbers for the IPO and are willing to take a temporary loss in the meantime. By the time they go public, they'll either drop the subscription model or it'll turn into what the Chinese providers already offer: basically just a cap on how much API you can consume. Same same.
It's not clear what API tokens actually cost them, but I looked into running a local model, and it's way outside the budget of an individual or even a small or medium business (hundreds of thousands of dollars). So my guess is that running these models economically isn't possible, even if they're delivering real business value (coding, research, etc.). In other words, at API prices I'd just stop using AI, and I suspect most other developers would too.
> It's not clear what API tokens actually cost them, but I looked into running a local model, and it's way outside the budget of an individual or even a small or medium business (hundreds of thousands of dollars). So my guess is that running these models economically isn't possible
Datacenters have massive economies of scale. Everything from cheaper electricity to having specialized, more efficient hardware to simply being able to run it continuously at near-100% utilization, all adds up.
Many things in the economy - most notably, manufacturing of most consumer goods - only makes economic sense once you're producing for/serving millions of people. This is not unusual.
> In other words, at API prices I'd just stop using AI, and I suspect most other developers would too.
Many say that, but I sincerely doubt they'd actually follow through. People might get more conservative about how they spend their tokens, but AI today is just too good at eliminating drudgery and boring / bullshit parts of daily work to give up on merely 3-5x price increase.
Sure. Issue is, no one is providing on how much it actually costs to burn these tokens. And as we don't know, we can only speculate.
> Many say that, but I sincerely doubt they'd actually follow through.
I have a $100 open ai sub and I track my token usage. Last month I spent roughly $2.600 in equivalent API usage. There is no way am paying that. I let my $100 sub lapse if next month I'll be using it less.
Look, I am not saying that there isn't a potential value out there. But the cost has to be bounded. If your opportunity is $1.000 and AI costs $2.000 to execute it, then you don't have a business model here.
> Sure. Issue is, no one is providing on how much it actually costs to burn these tokens.
You can assume Openrouter open-model providers serve at or above margin, because there's no branding so there's no reason to do it unless you can be profitable. If the Anthropic models are anywhere in that ballpark, they're very comfortably profitable on API.
> I looked into running a local model, and it's way outside the budget of an individual or even a small or medium business (hundreds of thousands of dollars).
That is a big exaggeration. You can have a perfectly usable local LLM setup that will power your agent for single digit thousands of dollars. Can even power multiple agents simultaneously, depending on the hardware and setup. Won't be fast and won't be frontier intelligence, but definitely useful.
Any model running on "single digit thousands of dollars" hardware will either be below SOTA (even for local models) or not even close to fast enough for real-time agentic work. Even the latest so-called "flash" models are large enough that doing real work usably with those on a lower-cost platform is at least dicey. You can fire off non-interactive work and do especially simple Q&A/chat (which is vastly more token-efficient than anything agentic - though even then latency will be high for anything genuinely SOTA) but that's about it.
Banks and health insurance are much more consumer friendly outside of the US, usually because of regulation. Turns out you can just tell banks “make transfers cheap and essentially instant” and they’ll do it, rather the bullshit they have in the US.
Another way Antropic misleads its customers is the description of the max plans. They are advertised as having 5x/20x the 5h quota as Pro. But the description says nothing about how the weekly quota scales, leaving customers to infer it scales the same way. But from what I've heard, the weekly quota is only 3.5x/7x that of Pro.
It's worked for online PvP gaming for a long time. Nerf stuff the min-maxers "earned" through game mechanics and sell over-powered "premium" things to everyone else to pwn them. Then nerf the old premium stuff and make new premium stuff. Forever.
I don't know if that's the actual origin of the term nerf, but it was the first time I'd heard it.
I think the origin of the word "nerf" as a verb came from the Nerf brand of toy guns. The idea being that "Nerfing" something is to turn it into a harmless version of itself.
Their fate is coming. Until the open-source models will be usable in machine with 256GB memory, they are done. Their behavior is unacceptable (Anthropic) recently but it won't last long.
Do Anthropic quotas give you precise remaining token counts or something? I have something similar set up for tracking my ChatGPT usage but it only gives percentages remaining, which is a pretty coarse metric.
Claude code supposedly has otel you can set via env. I haven't set it up, so I'm just repeating hearsay.. but it supposedly has everything relevant in it wrt token usage and cost
It's meant for their test env I think, so is not documented to my knowledge
It's obvious unless you have multiple requests from different models in flight at the same time, and the sum total usage comes out to less than a single percentage.
I dunno, I never sense nerfs for local models, but consistently a few months after launch for corpo hosted models, seems odd my internal model for the capacity of a model drifts for anthropic models but not local ones. I've been using LLMs heavily even before ada/babbage/davinci days, and trust my internal calibration over baseless handwavey explanations for why im imagining things, especially when I have data that shows capacity regression on frontier models for tasks, e.g. one shot success at loss, 0 success in 15 attempts once nerf is sensed. Others publish their quantified capability regressions which are also more trust worthy than this kind of handwaving.
But if it is placebo, obviously he wouldn't notice any placebo change for local models, since he KNOWS he is using an immutable local model. That comparison only works if he doesn't know what model he is using.
I suppose you have a point, there is room for bias in perception and it could explain my sensed degradation of service, however, it doesnt explain the failure of tests, which isn't tied to my internal perception.
I'd take this kind of benchmark with a grain of salt. At this point, I have a set of comprehensive guidelines covering both backend and frontend work, and for the frontend we go as far as explaining what we a good design is in our visual system, and even how to conduct a visual review when screenshots are handed to the model.
Deepseek 4.1 ranks very low in this benchmark but it has proven so capable that after being simultaneously on Max x20 and Pro x20 subscriptions, i've transitioned to using DS 4.1 as a daily driver and am very satisfied.
My point is, i think their overall ranking makes sense, matches my experience with out of the box capabilities for vague and underspecified tasks. But seeing a model rank low in their ranking does not mean that the model is incapable. Having skills and guidelines has a lot of influence on what you get out of a model.
I use the direct API from deepseek using Opencode, no problem to report, works like a charm.
Except maybe that the model often believes that he is running out of context, and needs to rush so i occasionally need to jump in to tell it that it still has plenty of room left.
But this does not degrade the quality of my overall experience in a meaningful way.
We could also use something that tracks concrete token amounts each tier gives you, in case they ever mess with it - and also maybe even the tokens needed to accomplish a particular benchmark, to see how much you can actually get done.
Do they use private benchmarks? Because if not, it could be selectively nerfed.
I also wonder if cache could be used to throw these off as well, where it's serving un-nerfed cache results for context windows that are identical to ones they've previously had for benchmark requests.
Seems like the only way to do it well would be to have some randomness involved that couldn't be cheated on - but you'd want to do it in a way that doesn't throw out the benchmarks too much, so your results can be compared still.
Thankfully, we can now use AI to design a test that tests AI. And the AI company can use AI to detect the test and cheat. And we can then use AI to implement anti-cheat.
All that energy wasted... could just drive a big V8 instead and make less money for the big tech
Often times people think of "nerfs" as my first prompt (which was greenfield - no or little code existed) used 5% of my plan usage. And then 2 weeks later (as the agent is busy reading hundreds of .rs and .ts files it previously generated) the user complains the usage is going down 30% for a single prompt instead of 5%. Attributing this to a "NERF" makes little sense because it's the same model.
If this is actually even close to reliable tracking, it's one of the most awesome benchmarks I've seen. I gave up on feeling the zeitgeist for what people were saying.
Theory (Conjecture? Hypothesis?): What we notice as "model nerfing" is the company diverting compute to training/running new unreleased models..
Remember that some people get access to the next flagship version long before us peasants do. I recall seeing the mention of "Astra" more than a month before it was officially announced
The more likely thing that would happen is that the provider begins silently interpreting (perhaps some) high effort-level requests as medium, etc., or having a classifier do this far more subtly. As such, the load on the cluster is less, and more resources can be devoted to training. Whether the frontier labs actually do this is purely conjecture at this point.
I assume there's classification going on where a really basic "Hi how are you?" style request sent to a high-effort instance can be routed to a lower-level instance. This... is pretty much fine with me, assuming they do a good job of it.
I would also assume they use nebulous labels like "Medium Effort" or "High Effort" map to quantitative amounts of compute allocation... and that these amounts can be varied manually or automatically. Right?
I mean, there's a reason why they call it "High Effort" and not "Exactly 5 Minutes of GPU Time on Exactly 10 GPUs." They want to be able to move those sliders and tweak those knobs.
The problem is that if benchmarks are run at a tight classifier that says "a request for high effort means check-under-every-stone regardless of simplicity" but a user request is run on a different classifier where "high means maybe high, maybe medium, maybe even low, even for meaningful tasks" then you're not getting the model that you saw in the benchmarks.
And, while you might be billed fewer tokens as a result (because the lower thinking would result in less investigatory work), you might not know this is happening, and know to dial up effort accordingly - you'd simply get a worse work product. And certainly, Anthropic's incentive for anyone on a subscription is to push this as aggressively as they can, so people use less of that subscription.
Sadly, I'd also expect that the OP's benchmark will be detected as a test of model capabilities, and thus be given a high classification so that this strategy remains undetected.
My guess is that they are dynamically changing the quality of the model to always keep the speed above some floor. So once it gets below that they switch to a worse quant or reduce reasoning level, or some combination of both.
The whole nerfing narrative puts in the spotlight now crazy supertitions come into being. The group think every day that everything is falling apart is crazy.
It's cheaper to run a quantization of a model, but its quality is reduced.
For example, if your weights were trained as 32-bit floats and you need 1TB of RAM, you could reduce that to around 256GB by quantizing to 8-bit floats. You also make the model faster in the process because there is less data to process to calculate the next token.
The game is to balance between the savings of quantization and making the model dumb enough the people notice
But as we also see in this thread, all evidence is dismissed and there's no good faith discussion. All data from the other side is lies and contamination. In that environment, it's power who decides who wins.
Or rather it's being.. Unsubstantiated. The Nerf conspiracy isn't that there have been a few harness and platform bugs leading to performance regressions, but that OpenAI/Anthropic have maliciously and unethically degraded their model performance post release to shed load and save money.
Yeah it's just inconceivable that companies whose entire business model started by engaging in wholesale for-profit theft and abuse of intellectual property would ever be so unethical as to try to lower their costs, especially just prior to an IPO.
It's at least highly implausible. Why would they engage in pretty uncontroversially illegal deception/fraud if they have so many other legal ways of gaming benchmarks, selling more tokens etc. available to them?
It's like arguing that your bank is scalping you by rounding down interest math on odd days of the month when they can just introduce a perfectly legal bullshit fee or otherwise change their terms to your disadvantage instead.
They wouldn't be breaking any laws at all. It'd be simply tuning their output as they see fit. And that's if you could actually prove it was even intentional, when in reality they have endless plausible deniability of 'oh it was just a technical glitch we've since corrected.'
Banks have far greater transparency and legal requirements. LLM companies are just delivering a black box that they have complete control over. And given the regular 'How's Claude doing this session?' stuff, it's almost certain that they're A-B testing various tweaks on a per session basis.
Again, these things are constantly measured. They sell HEAPS through their API access to enterprise consumers that expect a model to not be nerfed after it's released. And you bet many of those enterprises, some spending many millions each month, are measuring this shit.
So this is a case of extraordinary claims requiring extraordinary evidence.
And even though it's super straight forward to collect the evidence, there seems to be zero substantiating the vibe bro conspiracy theories.
My impression was that, at least with Anthropic, the point of my subsidized subscription is that I'll convince my employer to get an API account. If anything, the motive would instead be to ensh*ttify on the corporations already committed. Those of us inducted as boosters would get the fluffed up product.
Where people pointed out issues early and en masse, and Anthropic denied it was happening, gaslighted anyone claiming this was an issue, then begrudgingly admitted it was an issue, and then spent another two weeks "fixing it".
Anthropic is in a perpetual state of "oops, these 'bugs' degraded our model quality" and only admit the issues when it's immediately obvious and visibly affects a large number of customers.
Otherwise all open benchmarks can be (and are) gamed. And it's quite hard to judge the output of a non-determenistic black box that Anthropic (or OpenAI) constantly tweak.
If you're going to try to argue that companies doing things, completely legal mind you, to increase their profit margins is a conspiracy theory then you're not debating in good faith. Let alone when we're speaking of a subset of companies that were fundamentally built on wholesale unethical behavior carried out for profit. Let alone when we're speaking of companies who are all racing to IPO where short term results matter more than just about anything.
Another issue is also that the risk here is probably literally zero. Any evidence in support of such could easily be dismissed, with completely plausible deniability, as a short-lived technical glitch as opposed to intentional behavior.
> an explanation for an event or situation that claims a secret, powerful group is responsible for a hidden plot, rejecting the standard or official account
I'm sorry, but yeah. The official account is a harness regression and some platform bugs.
Where is the evidence they are underhandedly and unethically regressing their models to shed load and reduce costs? This is the conspiracy theory running rampant through the vibe boroughs; that they are bait-and-switching on model capabilities then "nerfing" them to save money and shed load. Where is the evidence?!
I'm not saying it's illegal, per say, so don't come at me with that straw man bull cock. This bro science conspiracy has been circulating for at least 2 years(I don't even know) and enterprises would certainly be pissed off if they were paying premium API prices for advertised and previously tested model capabilities that are suddenly under performing due to "nerfing" shenanigans.
I think the real story is just how easily people believe that purported conspiracy theory. It speaks to how little trust there is in these AI companies, and in Big Tech in general, that this "conspiracy" theory is perfectly plausible to lots of people
you can't explain honeymoon effects. like wow wow wow and then suddenly: same task, lesser performance is a misperception?
my fair lady gained a bit weight and the bjs lack variety?
[ I'm certain that that's a quote from some time ago by some commenter in a thread with a similar or even the same context ...but my Amnesia (Ｔ▽Ｔ ):･ﾟ:･ﾟ] fuck off.
you open two files, before and after you notice a nerf, and from worse comments to logical oversights, it's all damn obvious.
don't normalize this make believe bullshit and misleading people who you think barely understand what they see anyway ...
you wouldn't even know if models had somehow timed nerfs hardcoded into them, however much control over the stack you have.
There are a few bad puzzles (especially one regarding infernal critters) that one can use a guide for, but mostly it was a blast to finish it back in the day. Such a unique story by itself and then in a rare comedy / noir style on top! It's kind of crazy how they even came up with it.
I still remember the feeling of the ending, and the end credits music when the violin kicked in as it played the Manny & Meche theme! The music is at least as good as this game.
Political? It's using a Chinese LLM trait. If it only refused to talk about a particular kind of soccer, we'd probe it with that instead. The goal is not to discuss politics, the goal is to find out which model it is. What's political here is the language model.
reply