Just go ask some actual kernel maintainers who are being spammed by frontier models daily.
> These are relatively simple statistics.
According to Greg KH 50% of the vulns are flat out wrong, or the patches don't fix the problem, or the scenario is impossible in the real world, or just isn't a security issue.
The kernel has also been removing a bunch of old unmaintained drivers due to these reports.
I have to wonder how many of these 1000 paths are simply dead drivers with security vulnerabilities [1]. Impossible to exploit because nobody uses it, but easy enough to juice up the numbers of "found" vulnerabilities.
All the marketing from OpenAI and Anthropic about "we used the model to find X vulnerabilities in Y" without any technical depth whatsoever, as is customary in the security space, basically boils down to "big number scary. model fix."
I said in another comment that I await the technical writeup of the so-called new techniques that they aren't sharing for "safety reasons" but if history is anything to go on, they will never publish a writeup. They set out to get their headline and that's where the work stops.
To be fair that sounds like similar false positive rate to human bug reports, if not better. Although the agentic scale of these reports must consume huge amounts of time.
> don't see why it requires any people at all. It wouldn't even have to be done by OpenAI. Given a list of vulnerabilities
Who would put together that list unless individuals at OpenAI does it? It's not like we have machines that can think kind of like humans and do a bunch of stuff automagically.
It feels like this comment section has either gotten collective amnesia or didn't watch Greg KH's video on previous dump and grossly exaggerated claims from Anthropic [0].
From his talk, AIs can't reliably deduplicate or count vulnerabilities, dash for easy and meaningless escape hatches like malicious filesystems, and report the same things as found by everyone else. Indeed, this post literally says:
"Within 10 days while the scan was running, we observed over 100 Linux LPEs had been patched or had staging patches from other contributors, with ~300 more being publicly discussed."
A few ready takeaways: this is just hype. The community is already doing a great job at finding and patching the actual vulns without the attention-desperate big labs. And no, you absolutely do not need to pay them a million dollars each year for the privilege.
This behavior is the same thing that OpenAI has been doing to math -- produce tons of slop that are useless to humans. But this time, we don't even get to see how bad it is because they need to hide it to protect our precious little minds. We have got to be afraid though, very afraid.
Our field really needs someone like Terry Tao to unambiguously stand up to this behavior and say that they are not welcome to take such dumps on everyone.
> The community is already doing a great job at finding and patching the actual vulns without the attention-desperate big labs.
It is however reasonable to assume that a lot of bugs that are being found by the community these days were found thanks to those people using AI to aid them.
See for example what a PostgreSQL developer is saying:
> All software contains bugs, and PostgreSQL’s code base is large, complex, and old. Our git history – converted from CVS – now reaches back more than 30 years, and the first code was written a decade before that. There’s a huge amount of code, and that code contains a huge number of bugs. That’s always been true, and it hasn’t stopped PostgreSQL from being popular or successful. What’s different now is that it’s become vastly easier to find those bugs. […]
> […]
> […] a huge number of people on the Internet are now reporting bugs in PostgreSQL that they found using LLMs, and the PostgreSQL committers have been very, very busy fixing them […]
All I’m saying is, that it is reasonable to assume that a lot of other people also benefit from AI to find bugs in Linux.
The Linux kernel is also a huge amount of code that has been worked on by a lot of people for many many years.
There are certain types of bugs that are too obscure to realize are even there for a person alone. And there are types of bugs that even if you see that something is funky it is too time consuming to dig into for a human.
I am not saying that the Linux community is not doing great. They are doing a fantastic job, and have been for years, and they are giving all of that away for free too which is nothing short of incredible.
What I am saying is that I think AI allows everyone to find even more bugs than was humanly possible without AI.
It seems we are in agreement then. It doesn't cost a million dollar a year super-restricted AI to find those bugs and the community is already successfully doing that.
Meanwhile, support for Firefox from web developers seems to keep on sliding.
Whenever I have to fill out a new web form on a random site, there is about 50% chance that it will break. Regrettably and unsurprisingly, they do work perfectly in Chrome-based browsers.
I don't have a case right now. I usually report such problems to site maintainers while I'm at it.
But your comment motivated me to discover the "hamburger" > "Help and Report" > "Report Broken Site" menu. Will use it next time I come across a broken form.
Is it really 50% of the time you visit a site you have a problem? That seems crazy high for you to be encountering when my rate is approximately 0% across decades. Possibly you have an extension interfering?
I should qualify that 50% is the figure for new websites I've never used before. They typically belong to smaller non-tech organizations who probably outsource the web work and don't get around to fix things unless they are broken for virtually everyone.
My first reaction is to disable adblock and other extensions, but lately that has not been helping. I blame the lack of testing/support from web developers.
What kind of breakage is it? I feel like these days the devs would have to go way out of their way to create a breakage. Web standards and frameworks are so well designed & supported today that I'm baffled how they could screw things up so badly that they break it in one browser but not another. I could see it if they're getting into advanced features like direct hardware access (USB, hardware graphics acceleration), but it doesn't sound like that's what you're talking about.
Interesting. I didn't know if youtube started doing this for all videos, but this one shows only "featured" comments by default, so most of the negative ones are hidden.
Also, let's maybe not call the result impressive until we see what's under the hood.