Agree here. At my current place we have LLM apply deterministic rules where given thing cannot be checked statically. For example - all unit tests shouldn't access database. This can be checked statically. All comments should document why, rather than what - this cannot.
Humans tend to bring up things that matter to them + correctness, when they have time to think about it.
We've done similar thing as the OP, I think it will be 2 months now. Our processes were pretty focused on spotting defects beforehands already: we have standardised tasks defined with ACs following given/when/then format that we can write behaviour tests for, we have automated linting, static analysis, automated tests. Then there's also round of tests: both QA (LLM/human driven) and UAT (human). Since I joined we have added architecture tests to that mix, including static as well as LLM guided that look at how code is built and organised + writing documentation for what is being built / product rules (LLM driven). Looking at the stats - the amount of merged PRs has trippled, the amount of issues is decreasing, PRs time-to-merge went down to roughly a day (I don't attribute this particular metric to usage of LLMs in review process).
What I've noticed = good sides: LLMs are better than humans in spotting edge cases and reviewing code, given they have established conventions and standards (I saw PRS that I'd approve, but LLM was pedantic in reviewing a comment or class name); they are good in finding connections, ie I've had many cases where an agent built something where there was no guidance in the ACs and the reason was that it found related tickets in Jira mentioning given thing and filled in the functionality. LLMs are "cheap" to run = one can run review process for multiple PRs at the same time and forget while they run; they code defensively requiring developer to intervene - for example: I'm working on functionality where user is assigned to review a thing. Only one user can be assigned to review given thing at a time and there are checks for this. In last 3 PRs LLM is telling me - add locks because if there are multiple people reviewing given thing at the same time then the data will be lost. Yes, if we ever change business rules in this place this can happen. Finally LLMs are semi-consistent; sure sometimes they'll do stuff for shits and giggles following some convention I've not seen anywhere in the codebase, I ask what happened, and it tells me it picked it up from some tangentially related legacy place. Most often they follow already established conventions / adhere to rules in place.
Now the bad points: many times QA catches things that would emerge during planning or if developer would question what they're building or look at their work thinking critically (I find it that people are loosing the ability to question things) - LLMs dont help with preventing this from happening. People are loosing track of domain and product rules - in the places I've worked I was never just a developer, I was always a developer/something (forever to my dismay), so during planning or review I could say, from product perspective: this doesn't make sense, we don't need to build this. This no longer happens - there's no time where I can remember the rules that govern the product; PM often becomes the bottleneck of what the rules should even be. I'm also finding that people do not develop good taste - this piece of code that LLM used? Sure, looks good, go ahead, ship it. Path of least resistance is becoming a norm. It's also easier to create bigger PRs.
Anyhow - I'm old and tired, so sometimes I'm finding the LLM pros are greater than cons
You can annotate the method as `@return array` or even `@return list` (array of strings = indexed via anything that can be used as a key in PHP vs list of strings = indexed via incrementing integers, starting from 0). I guess the OP suggestion was to use Arrayable / Iterable interfaces, which would work as well, but would require more effort (= yes, declare a class)
final class Door {}
/**
* @template T
*/
final class Box
{
/** @param T $value */
public function __construct(private mixed $value) {}
/** @return T */
public function get(): mixed
{
return $this->value;
}
}
final class DoorInspector
{
/** @param Box $box */
public function __construct(private Box $box) {}
public function inspect(): void
{
$this->box->get()->open(); // analyser knows this is a Door
}
}
new DoorInspector(new Box(new Door())); // OK
new DoorInspector(new Box('not a door')); // error: Box given
You can even bind a template by a generic type with "of":
final class Door {}
/**
* @template T
*/
class Box
{
/** @param T $value */
public function __construct(private mixed $value) {}
/** @return T */
public function get(): mixed
{
return $this->value;
}
}
/** @extends Box */
final class DoorBox extends Box {}
/**
* @template T
* @template F of Box
*/
final class Shelf
{
/** @param F $box */
public function __construct(private Box $box) {}
/** @return F */
public function box(): Box
{
return $this->box;
}
/** @return T */
public function item(): mixed
{
return $this->box->get();
}
}
$shelf = new Shelf(new DoorBox(new Door()));
$shelf->box(); // DoorBox
$shelf->item(); // Door
What surprises me is that this is state of affairs for Google, a company with unlimited budget - to show you ads for things you expressed just a slightest interest in or have already purchased. And it's a state for a loooooong time.
Like - I just purchased some plants from Dibleys, a plant nursery in UK. I've gotten confirmation email to my gmail account. Now I'm served plant ads for that very same company. I'd expect that The Algorithm would be smart to decide "hey, here is a guy that is interested in plants, let's take a look at other people in his age group and occupation, what are they also interested in, and let's serve him ads for that. Or having failed that ads for other plant nurseries in UK". But no! It's astounding, that it's so utterly mindlessly stupid.
But you know what, actually I prefer that there's no computational effort here. Maybe this means that me being served a shitty ad for a company I like and already know will not attribute to global warming. Maybe this means that Dibleys isn't being charged for a clickthrough, but for impressions that cost a lot less. Maybe this cyclical stupidity is something that we're stuck with as a human race, because why the fuck should an ad serve some kind of purpose.
People complain about LLMs not respecting KISS, but fail to address why = LLMs being defensive and coding for situations that either do not happen or are unlikely to happen. That doesn't absolve you from deciding using your human brain if given change makes sense or doesn't. You can also tell LLMs to not program defensively.
Same with any other acronyms - if you don't set up guardrails on when given principle should apply you'll get code that you can only complain about.
My complaint is that LLMs’ approach to making code robust is something we used to pejoratively describe as “enterprisey”.
Oftentimes there are two broad ways to fix a gap in the logic. You can eliminate the source of it, which often involves refactoring but tends to result in less code and cleaner, more comprehensible logic. Or you can leave the gap there, but attempt to render it safe by adding extra logic to detect and handle the error condition. This reduces changes to existing code, but makes the overall system more complicated.
You can’t just indiscriminately favor one approach. Which one of those is best depends on the situation. The deciding factor is often something like a time constraint that you can’t know just by reading the code. You might need to talk to your team before deciding.
But LLMs don’t do that. They always choose option B: slap a bandage on it. Keep on adding more epicycles. In the 2000s we learned that, over time, always choosing this approach inexorably leads to codebases that are brittle, bloated and expensive to maintain.
So the problem, from my perspective, isn’t that we shouldn’t be coding defensively. It’s that LLMs encourage us to do it in a way that we’ve already learned is unsustainable.
Agree. It should fall on to developers to decide what to do = whether tackling tech debt is viable solution (whether making decision on their own or talking to people that can make such decision) or not. If, however, you're told "here is a codebase that you're seeing for first time, fix this error" and you simply cannot make the decision, even prior to LLMs you'd put a bandaid on it. Or at least - that's the standard approach = fix now, refactor later, where later in my experience never happens (sorry for this train of thought).
I don't think this is anything new, nor LLMs are to blame here. Although - LLMs want to make you happy, or maybe they don't think about what user actually needs, just how to accomplish the goal that was stated; they do stuff without asking questions (unless you constrain them or plan first). I've actively put in a hook for my agent to not make decisions on their own and ask me when it detects it's about to make a decision.
Agreed, it is not at all a new problem. The enterprisey codebases I’ve encountered are ones that have chronically had no clear owner, or constant ownership churn. I have been in that position before, and I also decided to make the code even more bloated for the same reason: I did not and was not going to have the time to learn the system well enough to safely do a better job.
I think where a lot of people run into problems with the KISS principle is they conflate simple and easy. More often than not, simple is hard. Which is maybe a soft reason why agentic coding, which is all about trying to make every task easy, leads so consistently to complexity and bloat.
Nobody else is likely to see this comment but I want to make it anyway. Yes, simple is hard. Thank you for articulating it so well.
And yes, that's also why LLM code is usually (unless driven by someone who really knows what he's doing) full of unnecessary complexity. Because although LLMs can get a much broader "understanding" of a code base than I would have given them credit for a couple years ago, their "understanding" is still shallow. And what you need to truly be able to achieve that hard simplicity (I heard someone refer to it once as "the simplicity on the other side of complexity", e.g. when Kepler figured out that orbits were ellipses, not circles, and that allowed getting rid of epicycles) is deep understanding. Which LLMs are not capable of, hence why their code is full of the kind of needless complexity that one produces when one has only a shallow understanding of the code. (E.g., you in that enterprisey codebase when you weren't going to be given time to reach deep understanding).
I don't think I agree that simple is hard. Simple takes effort. Simple requires that you have agency. Simple has all the information available. Simple means that you're accountable for something. I'll be cynical here - it's sooooo much easier to say: this is how it was when I arrived here, it's not my problem, what will I get out of this, somebody told me to do it like this, let's leave as it is, there's no point in touching this. And nothing in the above is LLMs fault! Well, unless at least now you can say - something did it like that! It's not my fault, it's what LLM told me to do.
I'm hopeful that LLMs make iteration and refactoring easier. That they'll allow developers to say to product (or even to themselves) "this module that is a dumpster fire? Sure, I can let agent investigate/refactor and that work that would have taken 2w will be done in half a day, do you want to go ahead?". That they'll allow developers to investigate how to make things simpler, more understandable, more coherent, to agree on a one fucking way to do things.
Anyhow, apologies for being glum and straying away from the topic, but I also wanted to make a comment that is unlikely to be seen :) the bile comes from past personal experience and is not directed towards anyone (it's more of an acid reflux).
When you say "simple takes effort", that's kind of what I meant by "simple is hard". It takes work, it takes looking at the problem for a long time (Kepler studied orbits for years) until you understand it well enough to have the epiphany that lets you reduce the complexity down to "oh wait, orbits are ellipses and that solves everything!". Since you preceded your statement that "simple takes effort" with a disagreement about simple being hard, I wonder what you thought I meant by "simple is hard".
So true! And it goes hand-in-hand with a different problem: You being able to try and brute-force solutions.
If you write your prompt as a request, and not carefully as an idea or suggestion, the bot can blindly agree to 'help'. And if you are stubborn about it, you can lose hours on something that cannot work, or make the code infinitely worse.
However, it's also the path of getting better at programming. To learn to take a step back, to ask yourself if it's really sensible. Just that now it also involves questioning the bot's help, as well as writing your prompts to prevent it.
> coding for situations that either do not happen or are unlikely to happen.
Not coding for situations that “do not happen” is exactly what creates vulnerabilities. But I also don’t think that such code is the main source of complexity in LLM-generated code.
An example to what I'm talking about: I have application that takes two inputs from request. I've put in validation on request values to deny values that are greater than 100. In my code I'm adding these values. LLM can say "cater for addition of these two numbers, because they might go over MAX_INT and result in overflow". This is a situation that won't happen. LLM can recommend - add assertions. That's unnecessary.
And the day you add a way to batch import request through another code path, you may forget the validation.
Having it in the lower layer is correct, a proper optimizing compiler should be able to detect there's a single code path to the function and that the checks have already been done, skipping the second check.
Arguably it’s only unnecessary when the value range is enforced by the programming language (types). The structure of who calls what in the code can change, and suddenly larger values can appear. You don’t want to have to perform whole-program analysis on every change to check whether all assumptions made in the past still hold. That’s also a source of vulnerabilities: assumptions that held when the code was written, but stopped holding sometime later.
That’s why I think code needs to document all its assumptions, and in a way that is checked by the compiler to the extent possible. If you document that the values have restricted range, then the AI should be taking it into account.
I vibe-slopped a small and local forms application for my family. I explicitly mentioned that fact to the LLM.
Yet, I removed 2/3s of the code, because it triple-checked that every file really existed and was valid, that the input form was less than a million bytes in size, that there were propper HTTP status codes for everything, that every possible error from reading data was eagerly checked and caught, and so on.
Like come on, three people will ever look at it. So keep it small and tidy so I can understand it better!
Google has a lot of subsidiaries all over the world that are very clearly NOT Google and never mind the fact that we only have one shareholder, whose name we unfortunately can not divulge. Employees are told on day one that they are NOT working for Google. They can never say they are working for Google, and they can't put that on their resume either. Google is to be referred to as 'the Customer'.
Tuike Finland is not a contractor. It's a Google subsidiary. Tuike Finland also has the auxiliary company name of "Google Technology Finland". Their official place of business is Ensontie 1, 49420 Hamina.
Don't think that I would be able to find information in the phone book whether given person is right leaning, or left; whether they said something racist or not, etc
The most hilarious moment for me was when my question that was 5y old was closed as a duplicate of one that was couple of months old. I provided an answer on how to work around the issue; the new question didn't have an answer, but it had more points. From what I read the original issue was still there. At that point I was rarely visiting the website either way, but the absurdity of the situation was staggering.
It's only absurd if you think the question being closed as a duplicate was calling you out for doing something wrong. This was how it intentionally worked: group everything, and provide a canonical answer. Duplicates were good, the idea being that more ways to ask the same question could lead more people to find the answer. The other question just happened to have more people find it (hence more points), but someone finding yours could then be redirected to the more popular variation of the question.
Humans tend to bring up things that matter to them + correctness, when they have time to think about it.
reply