Aside from bragging how cool they are, what value is there for demonstrating a yet-to-be-patched vulnerability? I don't understand why old-guard infosec stars from Calif can do as they want, responsible disclosure be damned.
Just as a reminder for some folks here: During their WeWorm research in July–September 2026, Calif didn't stop at proving RCE to Tencent—they spent an extra week using AI to build a fully functional, self-propagating zero-click worm across three live phones, triggering temporary account bans from Tencent during testing.
I don't see this as irresponsible disclosure. They show that the vulns exist, but don't give any other details (unlike previous Calif releases). Not even a hint at what the exploit chain is like other than that it starts with a PDF, which is a surprise to no one.
They're of course also doing it for internet kudos, but attracting attention like this will help encourage people to update. Thanks to this article, I'm going to apply the next iOS security update ASAP, and tell everyone I know to do this the same. (hopefully it comes out soon)
I do wish they clearly stated the vulns have had been reported to Apple and when.
We disclosed most of the bugs used in these chains to Apple, and are going to disclose the rest, if not already.
>Just as a reminder for some folks here: During their WeWorm research in July–September 2026, Calif didn't stop at proving RCE to Tencent—they spent an extra week using AI to build a fully functional, self-propagating zero-click worm across three live phones, triggering temporary account bans from Tencent during testing.
Not sure what exactly your point is. The WeWorm demo was done in a controlled environment.
Also we do this for free. We don't charge neither Tencent nor Apple. We don't even take their bounties. Why can't we have some fun?
If you're so upset about how we do it, my suggestion is go find your own bugs.
You literally just said your company released an exploit demo before disclosing some of these vulnerabilities to the vendor.
As for your suggestion, I was an infosec researcher, I worked for 3 years with vendors to help them patch before I can even tweet anything about my research. I even flew to their HQ to assist in patching when needed. After Covid I transitioned to a development role, and now I'm on the other end, handling security reports from researchers.
Based on this dual point of view, let me tell you that for a security engineer in Apple that will learn from your video about vuln your company did not even disclose yet, this karma harvesting is as far from "coordinated disclosure" as possible. Their managers will be fuming, and the stress they will be in given the media frenzy will be off the charts.
What is the value of revealing anything at this time point, when no patch is out? What use is it for the public?
Just last Black hat there was a "glimpse disclosure" that was later followed with proper fixes and details. I gave my AI agent the details that were supposed to be "useless", before the patch was out, and it found the vulnerability in less than an hour.
We shouldn't even enter this game. And with the track record of this specific group, I didn't exactly get the impression that their incentive is to help fix things, but rather to promote their brand while others be damned.
I gave the model my usual question of "I got a raise from X to Y, please explain what is the impact on my net salary". Answer was detailed and nailed all payroll edge cases, only to miserably fail in basic math at the end. It tried to calculate 430 minus 60 and got 410.
Comparison was done in the scope of coderabbit AI code review tool, which sadly makes it practically irrelevant.
My personal experience as a software engineer, and a former security researcher who did manual code audit, is that this code review tool has such poor results that it isn't worth the "noise" and friction it causes developers during C/I code review
Yeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.
At my team we added AI review because seniors were tired of always commenting on the same things, mostly because not everyone uses AI properly to produce production ready code. We tried coderabbit, qodo, and they all produce an amount of noise that end up being useless. A colleague and I then decided to create an internal solution that we ended up publishing under Revix AI. We have multiple control systems to avoid noise and only comment real issues, and we prioritize standards existing in the repo over llms knowledge. This one works so good for us and I don’t understand why the others don’t work like that.
Thinking of AI-generated code and AI code reviews as the same "AI" is not correct. The reviewer is using a fresh context window with no previous knowledge of the changes. That is why it's powerful, because you get the agent to interrogate the code without any preconception about the changes.
I found the Devin reviewer to be very good, and have heard good things about Cursor's Bugbot. I've also found asking an agent with fresh context or subagent to adversarially review locally is good.
I’m not saying that it’s not useful, I’m saying that it’s not useful in a “human in the loop” situation. This type of AI-to-AI review should be done agent-to-agent, not through Github PRs with tools like Devin.
In a manual review, I then expect all “machinery” to already be properly reviewed, and can focus on design / architecture. I would like an AI assisted review tool to make that part easier, not do the actual review for me.
I’m not convinced by the automatic agent-to-agent thing. I find that, if I manually ask a standard harness “Review a..b, individually and for combined effect”, I get some mix of catching genuine errors (some quite deep), incorrect flags where the correct course of action is to ignore them or modify the commit messages, and comments where the correct course of action is to think deeply.
If I were to automate the back and forth, I would get spurious changes that “fix” what wasn’t broken and a removal of the actual interesting bits.
this touches one of the reasons opus 5 came by default with 200k context and 1M gated behind usage tokens, auto-compaction by default, and it keeps telling you to clear and start from scratch all the time, AI is does a better job fresh, telling multiple agents to pipe around ideas without the human intent is the worse thing you can do
Specialized AI code review software is so pointless though. Back when agents were dumb about git surgery and tool use it might have had a purpose, but now you could replace coderabbit with a skill and I bet the results would be better in some cases.
Mechanical code review by something like sonar qube is much cheaper. Use that for low bar quality gate and AI after.
AI code review is startlingly effective. Continually finding things me and my colleagues never would. Well the decent models do. Maybe not so much the cheap ones.
The noise is a huge problem, indeed. Still, a panel of review agents using models and harnesses different from the one implementing a set of changes has proven immensely useful for myself. The panel is basically an n×m matrix of agents and highly specific review prompts, i.e.:
- review for intent fulfillment: is the ticket done?
- review for correctness: race condition bugs, ...
- review for security: check against this list of sources and best practices
- review for api conformity: identify all surfaces of systems outside this codebase touched by the code changes and check against their docs
- etc. pp., same for maintainability, observability & analytics, test coverage, usage of feature flags
The matrix is sparse, so not every model is used with each of the review categories. Effort levels vary, too. The next stage does a consolidation across all findings, then another stage spins up one agent per finding and investigates the whole codebases for identical / similar instances of the finding; finally, it suggests a fix.
This works extremely well for finding deficits, but the amount of noise drives me insane, too. Lots of feedback is technically correct and "by the book", but pretty useless in practical terms – or even detrimental because the amount of code written and thus the size of the change set explodes. I'm not yet sure how to tackle this problem, any suggestions are welcome!
You need to literally review the review with another llm pass to push back on the first. Ask it to do something like reassess the severity claims and only surface real P0 to P2 issues.
That's a thoroughly uncharitable view. Especially in smaller orgs with a minimum velocity dictated by the company's need to survive, the amount of code required to be written just to keep up with your competitors is massive. Trying to review that all by hand, thoroughly, is draining, thankless, and tedious. You end up with a few fast movers producing most of the code, and some slower movers forced into a reviewer role they never signed up for. It's an unhealthy dynamic.
Where I find it shines it to find inconsistencies. My readme or docs or ADR something should work like X but it finds a test where it tests something different and the test is green. Or other similar drift.
Yes, your prompt need to include to look for certain “quality” aspects you care about. But once that is there it can help find a lot of things.
It can also help in finding edge cases. It is really about the prompt.
All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly. I also still manually review code, and usually also catch issues, but the severity of what I find shrinks ever further as agents get better.
The sprawling code comments are becoming the most draining part of code review though, that's really killing me from the inside.
> All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly.
No, none of today's AI would give you enough signal around "should this thing be built in the first place" nor if it's the correct solution on a high level.
They don't understand why you are doing what you are doing, and even if you explain it, they still don't actually understand the motivation and lots of other things.
You'll get them to do guesses and pretend they actually know how to prioritize and will tell you it makes lots of sense, whatever they come up with. But try following it blindly and you'll see where you end up.
This is why "one agent + one good developer" beats "thousands of agents working in a swarm" still today.
I don’t think I claimed agent reviews to be a panacea. It’s a tool that can help you lower the review pressure in companies working with agentic coding tools.
> What really important things are human reviews catching in your org?
Another person said:
> 1. Whether the thing should be done in the first place
And you replied:
> All of these are angles an AI reviewer can test for as well, and will (IME) mostly catch mistakes correctly
Which as I noted, is very far from the truth. I neither claimed that you said "agent reviews are a panacea", but when you claim "AI can solve all those things" and two of the first items cannot be addressed by AI (today), then I'm rebuking those specific things, not some other general point you implicitly made.
It is mostly useless at figuring out if "should this thing be built in the first place" and "if it's the correct solution", and mostly cannot help at all with those things.
Where "mostly" means kind of what it says but also not really.
The AI review are still quite far from having the same level of critical thinking and high level knowledge of your application, what you have done in the past and want to do next etc.
If you don't master this for your own project, what's even the point of your job.
They do catch important things but it’s really contextual. You can’t grab a model slap it on top and say code review . Hence a dedicated review tool is almost dead . Code review should be part of your pipeline and consume test results from the original task , open spec etc . If you do not have that code review will not help if you do , what is the point of task rabbit just slap review against
Just as a reminder for some folks here: During their WeWorm research in July–September 2026, Calif didn't stop at proving RCE to Tencent—they spent an extra week using AI to build a fully functional, self-propagating zero-click worm across three live phones, triggering temporary account bans from Tencent during testing.
reply