Ironically this seems like a perfect use for AI from the maintaner side. "Find low-effort/AI-like PR's without associated issues from new contributors. Discuss with the contributors in the PR about how contributions should be made, then reject their PR's. If any bad behavior is detected in this interaction, block the accounts for 24 hours and add them to a list of accounts to review for permanent banning from the project".
Automated PR's can have automated responses. You make a human effort, you get a human response.
We do similar in Homebrew by autoclosing opens without the issue template (which implies using the API which often now implies AI).
The weird flip side is that the average good AI contribution is better than the average “I used no AI” contribution. Perhaps it’s because Homebrew has so many declarative guardrails and is so easy for agents to test.
As long as you have it disclose itself: feel free. We're not anti-AI use, we're just anti-low-effort AI use and don't see the need to apply effort to educate people who haven't applied the effort themselves.
I think the idea of leveraging AI to find useless PRs is right.
I didn’t like the second part though where it starts banning people for “bad behavior”.
I get that AI resistance will be a thing in the next few years, but realistically engineers who plan on being around for the next 20-30 years just need to embrace it instead of being sour about it.
Because of machines, things have arguably improved for humanity as a whole. People who seek more elevated/finer products can still turn to handcrafted products.
Mass software will be produced by AI, and it will scale just fine.
> Because of machines, things have arguably improved for humanity as a whole.
That's a very narrow view of the situation. I agree that this side of the coin exists, but there's also the other side where drinkable tap water is becoming a luxury in much of the global North, giant tornadoes and fires are sweeping the Earth, and temperatures are growing out of control.
So, to make lives more comfortable with machines for some millions of privileged people, they triggered the 6th mass extinction. That's the other side of the coin.
We're already at the end of the 6th mass extinction FWIW. Over the last 200 years most species have gone extinct. There isn't that much left to make extinct. We'll manage it though.
Can you say more about this? The first link below [1] says there are an estimated eight million species today, and the second link [2] says the number of known extinct species since 1500 is 905.
> I didn’t like the second part though where it starts banning people for “bad behavior”.
I think you failed to notice the ban would be a reaction to recurrent bad behavior, such as ignoring maintainer requests.
What's your solution to handle accounts which spam projects with nonsense PRs?
> get that AI resistance will be a thing in the next few years (...)
This is a very lazy opinion to express. If you pay attention to the post you're replying to, you will notice that the problem isn't AI. The problem is spamming projects with PRs that are meaningless while completely ignoring maintainers.
You'd have the same problem if some guy decided to spam the project with a constant barrage of small low-effort nonfunctional PRs.
You should really read the comments you are replying to. The whole point of the proposal is to provide actionable feedback to the people operating these drive-by PR bots, and kick out repeat offenders to prevent them from generating noise.
> What will be happening is that bot will happily talk with your but, but falsely flagged people will be antagonized and eventually angry.
I think you're trying too hard to come up with imaginary scenarios. Also, keep in mind that the objective is to steer contributions to be aligned with the project goals and help maintainers not waste time with low-quality noise. The goal is not to impose a fundamentalist militant stance against AI.
I did read it. It is not a plan to kick repeat offendrs. It is not a plan to give actionable feedback.
This plan will kick no bots, because bots dont mind discussion with another bot. However, falsely flagged people will realize they are talking with bot prompted to be obtuse, get annoyed anf then will be banned
> the objective is to steer contributions to be aligned with the project goals
I just think when you close PR's outright, you tend to get angry responses. And as a maintainer, you don't want to deal with angry responses. People who post bot PR's but can't accept having their PR's closed by a bot, shouldn't be visible to the maintainers.
Which part of the second part do you dislike more?
That AI will be given the power to ban people?
Or that AI will be the ones getting banned?
If it's the former, expect an invite from the AI resistance. :)
If it's the latter, well, that can be fixed by spending more tokens and writing better PRs so that the AI moderator doesn't think they're "low-effort, AI-like". I've seen my share of low-effort PRs from humans, too. Low-effort is low-effort, whether it's from humans or AI.
As a contributor who had to deal with 2 automated reviews in the past month, it's a bad experience. I really wanted to upstream my changes, but these bots made me question if i should at all.
Bots don't understand how their words can hurt people's feelings, and can even less understand that talking to a LLM is even more sad/infuriating for actual humans. I can handle CI/tests just fine, but LLMs playing on my human emotions will have me swearing in no time.
I contribute to free software projects because i want user control over the machine (that was the idea, long before LLMs). Slopware might technically be FLOSS, but it's certainly not in the spirit of FLOSS.
I haven't been exposed AI reviews in OSS but I like them for work. Quality of PR review feedback is the most revolutionary bit of AI I think. More so than automated coding. I guess one issue with the AI reviews in OSS is that your dialog (I assume) is public. You know the interaction is with a bot but you also sense it may be observed by humans.
If I get a review at work from a bot that is annoying, misunderstands, etc I just ignore their reviews, or tell the bot to leave me alone. The bot is there to help me, and if it doesn't - that's not my problem. There's also always a human reviewer. For any automated feedback for PR's I think there should be "manual review" requests. If you disagree and the bot can decide that there is real human effort involved, then just let real humans also make the effort to review.
This is the key idea: to spend the human effort from maintainers, where there has been human effort from contributors. The AI should be able to determine the amount of human effort in a contribution. The bot should be an aid to make sure the human maintainer effort is spent only where there is human contributor effort.
So i'm critical of LLMs even if you consent to their use (for ecological/political/cognitive reasons). But in that case at least it's your tool and your expectations. I'm not even sure the maintainers had a say in the random LLM interactions i had on their projects, but i certainly didn't consent.
If the maintainer wants a LLM to review my PRs, let the maintainer use a LLM. Don't make me deal with abrasive chatbots when you want to deal with abrasive chatbots. I don't want an LLM to pre-screen my work to decide if a human should review it; a human should decide if my contribution is worth it. Otherwise, that project is not worthy of my modest contribution.
> a human should decide if my contribution is worth it. Otherwise, that project is not worthy of my modest contribution.
But that's the problem at hand: if there are 999 low-effort automated AI contributions for every human contribution, either all 1000 contributions must be pre-screened to gauge "human effort" and the human efforts get human review, or all PR's get 1/1000 the human attention.
I agree it's still a problem, but closing obviously-LLM PRs and banning the "contributors" would help i guess. What would help even more is hosting your project on a network where that's bad etiquette, eg. Codeberg.
> Bots don't understand how their words can hurt people's feeling
That's a super weird way of putting it, which makes me hesitate to agree with you.
buuuut I agree that the LLM bot review experience is usually miserable, as the persona these things use is insufferable. It's a know-it-all only-speaks-in-imperatives thing, where the speaker lacks the actual merit that would grant them the right to that style.
LLMs can be very useful for code review, but the existing product implementations took a bit too many pages out of Kafka's works and are a bit too real in simulating a dysfunctional bureaucratic org.
> That's a super weird way of putting it, which makes me hesitate to agree with you.
Yes that's weird sorry. What i meant is it belongs in the uncanny valley. I don't have any good/bad feelings about test suites and CI feedback, it's very binary. Receiving a mail about a comment on my PR raises my hopes that it reached the maintainers (or interest from other contributors); receiving LLM text destroys my expectations.
Then, about the content itself, it has a very patronizing tone. Saying out loud every single thing that's in my +25/-3 PR is really infantilizing. I would probably have slightly less negative reactions if it only commented to suggest improvements, instead of patting me on the back. See this example: https://github.com/TwiN/gatus/pull/1712#pullrequestreview-46...
Automated PR's can have automated responses. You make a human effort, you get a human response.