There's an anecdote about curl, but I'm not sure if there's enough in here to draw conclusions from. 20% of vulnerabilities have been made public, and many of those are marked as "withdrawn" or "duplicates". However, it's unclear whether the remaining 80% have not been made public because they're not fixed or real. In other words, this is analysis of survivors (public); not much can be said about those still under embargo.
I'd expect the remaining 80% are delayed not because there's something statistically special about them, but from the massive workload thrown at humans in the loop.
Anthropic needs to teach their systems to propose a fix too. Detecting a coding problem and proposing a patch is what we should expect from a company with supposedly the best coding LLM.
> Anthropic needs to teach their systems to propose a fix too. Detecting a coding problem and proposing a patch is what we should expect from a company with supposedly the best coding LLM.
Of course it's trivial to have an LLM generate a fix, too. They probably do. It doesn't mean a project (1) wants to take an LLM-generated fix at all, or (2) can immediately commit an LLM-generated fix without review (which takes time).
But right now they’re not even offering one. They’re just putting the burden on the project without even trying to help fix it.
They have all the resources in the world, and they won’t even try.
And that’s of course assuming it’s a real bug, as we’ve seen a ton of them don’t seem to be. The last time we went through this, curl reported not only that a bunch of them weren’t bugs, but some of the bugs were listed previously on their mailing list, and that’s probably where the LLM got them.
So not only is it finding small bugs. And nonbugs. It’s reporting previously found bugs.
Pilling work on maintainers. Over and over again. To help? No. To make big numbers to show off for their IPO documents.
> A good amount of the patches is straight garbage. It's not helped by the fact that a good amount (20~30%, maybe more) of the issues is also junk themselves.
Technically they are, but they could try harder given their capacity. Maybe triage so they could use more tokens to create patches for critical bugs first. Right now they're just aiming the slop firehose at humans.
Could also be that most of the findings are low signal noise. Even the batch that was sent to curl was one medium and 39 low severity. One can only assume that the vast numbers of findings that haven’t been sent are even lower severity.
I'd be careful to take an examplary projects like curl and assume that level of quality is the average or even close to it.
Compared to curl, even the Linux kernel is garbage, and the Linux kernel is probably higher quality than all of the enterprise software that runs the world.
Claude is perfectly capable of fixing most vulnerabilities; if that's what maintainers want, they can just ask for it. If the concern is cost, Anthropic offers a Claude for OSS program.
The reason Anthropic isn't proactively sending patches is, I suspect, that those patches would not be welcomed, because trust in the models isn't currently at the point where maintainers are willing to merge them without human review, which takes approximately as much effort as writing the patches themselves. You can imagine the wailing and gnashing of teeth about "slop PRs" and "extractive contributions" that'd be all over HN.
Anthropic's oss-scanner includes patches. We've received a batch and I'm working on some of it.
A good amount of the patches is straight garbage. It's not helped by the fact that a good amount (20~30%, maybe more) of the issues is also junk themselves.
I'll hopefully be able to share numbers (or data) after we're done with the batch.
I won't claim that my experience is representative, but what I've seen is closer to 50% being false positives, and then maybe 20% of the remainder being bugs but not vulnerabilities (aka lower priority bugfixes). The suggested patches have a much worse hit rate though, to the extent that I accepted only a small fraction after reviewing.
That said, an automated system generating only 30% true positives is probably still ahead on ROI as long as we're talking about tens of findings and not hundreds. The amount of human time spent investigating things that turn out to be false positives being the main difference for how over or underwhelming it is.
Yeah, I was being "optimistic" in my estimate. And of course with a batch, you start at the high severity end, so it gets worse as you progress through. (Well, I assume at least, even some of the "critical" was just plain "not even a security issue" in our case.)
You're assuming the other 70-80% is actually good. It's not, there's a whole spectrum between "junk" and "good". A whole bunch of this stuff is "technically correct, technically even a security issue, but also just plainly not of actual importance".
You get free meals, 70-80% of them tasty on different degrees, and the rest burnt. That's OK because free meals, right? Except the small letters say you have to eat 300 meals a day.
> Claude is perfectly capable of fixing most vulnerabilities; if that's what maintainers want, they can just ask for it. If the concern is cost, Anthropic offers a Claude for OSS program.
So, "Here are hundreds of bugs. We won't show you how to fix them because you probably don't like AI patches, but you can always pay us to get AI patches. Come on, I'll give you six free months if you qualify."
And on top of that, it's a bit negative. How many millions of dollars is one perfect-10 scoring CVSS finding, total destruction style RCE, found in curl worth? Considering how many people use it.
It's worth zero millions dollars to curl. It might be worth something to attackers, but "wide use" is not the same as "the ability to compromise everyone" because most of the time, curl isn't used to automatically fetch untrusted URLs in a way that's controlled by the attacker from an environment they care about. It's worth much, much less than a 0-day in an iPhone or PHP. You'd need a buyer with a very specific itch to scratch and there won't be a bidding war.
By extension, if a vuln is worth something to attackers, it's probably also worth some fraction of it to their potential victims. But if you're an interesting target, your defense can't be based on the ability to find out about every single software defect ahead of the time. You need layers. And if you have that, the importance of proactively acquiring every 0-day is diminished.
A client-side bug in a command line utility that is mostly used as a debugging tool or glue in shell scripts where the URL is not attacker controlled? Probably close to nothing. Maybe if you can find some other software which lets you run curl with an attacker controlled URL, you could package it up as an exploit for that piece of software, but it's not a common config.
The most likely explanation is that it isn't worth submitting a massive stack of bullshit to human review.
If they actually were sitting on a PR gold mine of vulnerabilities they wouldn't be throttled on review resources while artificially inflating/misreporting their numbers. They would be diverting resources to clear the review queue.
when i was in the 5th grade (give or take a couple years), there was an innovation project where we were to bring in and present to the class an invention.
i remember a girl inventing what i now know to be a "flexible weighted dip tube" – think of it as a shampoo bottle that you can pump upside down – and was amazed.
like this submission, the flexible straw wasn't really invented by the student. at best it was independently discovered, but this patent was from at least twenty five years before: https://patents.google.com/patent/US3547296A/en
to that end, i wonder what the point of presenting these inventions as your own are – especially in the time of the internet when prior art is so easily discovered. or, did i miss the point?
> i wonder what the point of presenting these inventions as your own are
Because if you independently invented it then it is your own.
The value to the student is not necessarily that the idea is novel to everyone else but that is novel to her. And of course the process of invention and development is something that many people never experience, and is valuable in itself.
I invented a steam engine when I was in 5th grade. It wasn't completely novel nor efficient, but I invented it independently, and my science teacher showed enough interest to spur a 50-year warm memory.
Praising kids for achievements is a net positive. They use it to focus their energy into future growth and development.
The article is science-reporting trash, of course, but praising the kid is a good thing.
They're either dishonest, or they're too stupid to know how to use the internet. You pick. Seriously, what's the alternative here? The only exception would be an honest person too afraid to speak up, to correct the dishonest/incompetent people, and I could agree with that.
I don't blame the kid though. Looks like they did a great job on it!
I'm willing to look the other way when N=1, and 0..n is a small number of values on an indexed column. As others have pointed out, sometimes it's difficult to merge two queries like `get_item_list()` and `get_items()` located in different parts of the code, especially if they cross service boundaries. It's ugly, but may be a completely tolerable situation as long as timing profiles and usage patterns show that it's not going to blow things up. So there, 1+N => 1+1 which isn't inherently scary.
It's when 1+N turns into 147 queries, one for each line of a table displayed on a web page, that it really chafes.
That, is also a thought terminating cliche. Sure we want to choose what we spend time on, however, we don't know what experiences will meaningfully contribute to our lives. It is only after we have gone through them that we realize which ones were significant. And these vary for everyone.
We are going around in circles. When we are doing something, we won't be able to do a thousand other things we could be doing. Only the future can tell whether that turns out to be great or not, however, how we react to those experiences is under our control.
Millions of people live about the way you have described, doing manual chores (perhaps out of necessity), and may be they are missing out on the joys of commenting on HN or may be I am indeed missing out on something when I eschew some degree of manual work (I have experienced this).
The point is not automating stuff, it is what do you do when you have automated the boring stuff that is matter - do you just scroll your feed ? Not to mention that some amount of boredom is necessary to make us value the interesting aspect of life.
OP is more correct than you realize. Your judgement of how life was is in the past, while what your life is in the now.
umm, its like the hallmark of adulthood to learn that life is a lot about doing things you don't want to do. There's obviously a spectrum. And I don't mean forcibly through coercion. I mean yeah it sucks to wake up early and clean up and get your exercise in and be accountable to your team or take your kid to basketball practice or feed the cat. Or go out your way to pick up that trash someone else threw out, or let someone go ahead of you in traffic even though you were there first or take the stairs because someone is moving in today and needs the elevator more than you or take your dad to dialysis and listen to him mention the same things 5 times, there and back or not just eat anything you want whenever you want.
I don't know it sounds like you're advocating for doing only things you want to do. Automate everything else. It's all philosophy at this point. Life doesn't really work that way "and it shouldn't" - where shouldn't is wholly in the philosophy category, of course.
You left out washing your clothes in the river, carrying water from the well, butchering a chicken for dinner, walking ten miles uphill both ways and a whole lot of things that you seem to have automated away.
haha you're right! That is the entire nuance of the discussion. There's a line somewhere in there. And it's not as pedantic as it seems. I still do cook my own food. I don't grow it and butcher the meat. But thousands of years of cooking people still cook food. (and not me but people do still grow their own food).
Efficiency and automation isn't some objective linear path toward enlightenment, that's not unreasonable, is it?
And you might say "people cook because they want to". Not exactly given the whole socioeconomic realities. But also "want to" is nuanced. Cooking is a lot of work and there's many parts of it annoying. It's worth it.
reply