What a strange way to put it. Market forces are a good thing in a market economy. Only someone who secretly wants or hopes for monopolies would you say something to the contrary (a VC).
The party has done something about enormous involution in solar panels, for example (https://www.csis.org/analysis/chinas-solar-industry-upheaval...) and previously steel. They're planning something for cars. They don't on LLM because of the newness and the wish for preeminence.
Pouring 100% of your tech optimism, and probably portfolio, into a product that some say is about to get Wile E Coyote flattened by market dynamics would probably inspire serious market skepticism.
China has a different perspective on it, they believe that there is such a thing as harmful competition and they are willing to step in to stop it.
Enshittification and related problems can be a result of market forces just as much as they can be a result of monopoly/duopoly or a small cartel. Excess competition sometimes results in all firms scraping the barrel to squeeze out pennies, especially with technology (such as large online marketplaces) making pricing more transparent.
Marx actually predicted that ever-intensifying competition would destroy markets through overproduction, although he did not use the term involution.
Price wars are good even if they lead to more bad products on the market. If quality is important people will pay more for it and ignore the bad ones, if not then everybody saves some money.
The reason it doesn't work like that IRL is centralized marketplaces. If the winning strategy on Alibaba is low prices, bad quality and botted reviews to compensate then every seller has to do it to survive, because they can't get buyers outside the platform. That's not excess competition. It's a lack of competition just on a different level.
> The reason it doesn't work like that IRL is centralized marketplaces.
Well yes, that’s why I mentioned those specifically. But even if the marketplaces were split up, someone could create an aggregator to comparison shop and the same effects would apply. The problem for producers is that the Internet erases information asymmetry.
It’s also not just affecting low end goods, it’s a constant pressure on everyone, which is why many formerly upscale brands are seeing the same problems. It’s also a general problem with public companies, as large shareholders demand constant growth, as well as many private companies owned by PE where brands are stripped for short-term profits.
My experience is that the best low cost mid-tier products right now are coming from fronts like Vevor and Fanttik who do sourcing from noname factories in China. I’m not sure if their position is sustainable; it’s not like they have much of a moat. (I guess Fanttik has a team that adds some slick design to their otherwise utilitarian items.) If that model holds up then maybe that’s the future, but I suspect that they just have a temporary advantage thanks to a dual presence and connections in both the US and China.
> Platform teams are engineering-led rather than product-led. There is almost never a product manager handing you a roadmap, no revenue line to follow, and no market to lose.
It's precisely because of this framing and mentality that platform teams don't actually serve people well and are usually highly dysfunctional towers of people inventing work.
The fix for having no market is to act like the teams you serve could leave. This whole article lists signals, and none of them is that. Being captive does not mean the users or internal teams don't have other options and don't notice. Being product-led means caring about your users. A platform teams should be product-led, not engineering-led in that very narrow meaning. Otherwise you invent work as this article so wonderfully exposes.
You're assuming product people don't 'invent' work, which is faaaar from true. The more technical the area, the more nonsense 'product' generates, that engineering then has to either redo, clean up or fight. The 'product' people on 'product-led' teams also generally own the internal comms and marketing (internal) side of things, which often has the effect of silencing engineering (and many other negative side effects). (Not always but often imo)
Engineers should understand their customers and their product (whether internal or external), should understand their metrics, and should be able to make intelligent, informed decisions. Sticking a non-technical 'product' (aka marketing) person into the mix is generally net-negative imo.
I think the problem is one of management and prioritization and decision making - when multiple engineering teams disagree, how do you decide? Management consistently hires people that solve management problems, and mediating disagreements or picking product direction are core management functions that have been completely left behind as managers become purely MBA number people.
I don't think they're making any claims about non-platform teams never having similar work-invention problems.
They're just saying a good platform team engineer cares about their users.
Which is fairly at-odds in spirit with the quoted idea of "there's no market to lose."
The original article does say "talk to your users" but it also de-emphasizes this by having it among an apparent laundry list of other signals: crashes, costs
Focus on cost without talking to your users - aka the people who care about the spend? You might spend a lot of time reducing a number that isn't very important right now. Focus on crashes but don't talk to your users? You might address some things with easy workarounds ("i hit retry") while ignoring much more painful toil. This is captured in the details for those sections, but not in the headlines.
There's also some interesting stuff in there in some of the bullets, like overloaded use-cases and partner-to-prototype, but again, that's just more specifics on how to talk to your users.
I think the article would be a lot more helpful to a lot more people if it was a "Guide to Talking to Your Users" and then framed each of those specifically as: user discovery, and how to talk about the given part.
Previously I worked at a very large company, my team was mostly isolated form the rest of the tech stack of that company.
Suddenly came a mandate to try to integrate our systems together, the first goal was to use their authorization system (which involved, I kid you not, setting up 3 separate EKS clusters that talked to each other and if the main one goes down, all go down) that the core Platform team of the company was setting up.
Worse even, the plan was to use us as "guinea" pigs for their new systems before rolling out to the rest of the company because our team was smaller and therefor wouldn't be as impacted by problems on their side.
I was vehemently opposed to this, all our other experiences with their stuff was just a huge amount of pain. I remember clearly a meeting we had where I said: "Why would I use your stuff that is not well documented and that I don't understand when I can use open source stuff that is documented and that I can understand". Their only argument was that said they would have internal support for us.
I came up with this concept that I tried to push on to them, that their stuff shouldn't be this monolithic tower of babel monster, but instead should be a buffet where I can pick and choose what to use. Our requirements were very different from their stuff and I didn't want to run into problems because of stuff that had 0 benefit to us.
I ended up leaving that job mostly because of this.
I've had similar fights over the years, on both sides of the discussion.
"If our internal design system is barely half as well documented than Bootstrap then teams will still use Bootstrap. We should take a page from Bootstrap's documentation and include as much detail as we can, on everything."
"As a platform team, your product is this API my app calls. My beta environment should be pointing to your Production, never your beta environment. If that means you need to support better multi-tenant from the same 'app', then you need better multi-tenant. Your outage impacts my testing, which impacts my velocity, which impacts my deadlines. Your backwards incompatible updates need to be planned and scheduled and rolled out like a Production update, every time."
Product mentality is still very useful in platform development. If you can't sell your platform on its documentation and its stability and you must sell your platform on mandate and top-down control, you probably aren't doing as much to help your engineering culture as you think you are.
Yeah, you get what I was trying to get at. It also helps to think about open source (and even paid) alternatives as competition.
Although my main problem was that they were designing these very generic solutions (that were both super complex and very brittle) that can fit every scenario when my _actual_ scenario was very simple. I couldn't get my point across that I didn't want to buy-into their full platform because of risks and complexity on my side, but that I would be open to buy-into small solutions if I saw the need for them in our product.
In any case this company platform team had this mentality that the company should operate like google, trying to build these massive all-in-one solutions that everyone under the company should use. When in practice it would never work because of the sheer amount of people required on their side to pull that kind of initiative off (on top of other considerations like ROI, downtime-risk, etc).
This sucks. Our team strategy in these situations was to just wait and hope the mandate goes away. Or I'd personally de-prioritize that stuff. Meanwhile a couple of us supported our smaller team's platform.
Things never got too bad for us, but they easily could've, and it'd probably be like your scenario where I quit. Was lucky enough that a couple of big layoff waves didn't impact us but did drastically reduce the amount of bogus technical mandates coming in.
Somewhat true. Platform teams have to be engineering & product led. I run a platform team and we work very closely with our customers to understand what problems they are facing and identify platform level solutions for those problems. But we are still hampered by the team not using the platform for their everyday work, reducing the empathy they have towards its papercuts. However, being purely product focused can lead you astray of your users. Our PMs invent capabilities that no one really asked for ignoring the rather large backlog of capabilities that they already need. That's also bad.
The problem with every platform team is that, impact is hard to nail down. Does X using your platform Y to generate revenue mean your platform Y has indirect impact same as X? Most companies don't think so and end up destaffing their platform teams. Until, the destaffing ends up in much more inefficiency because everyone is inventing their own crooked wheel, spending time on the same capabilities and detracting from product development.
Not really. Not unless your PM is also an extremely deep technology person. Here is a simple example from our current debate: should we build the resiliency capability that hurts multiple customers and who have asked for it because it reduces their toil or the AI feature that nobody has asked for that our PM wants? The AI "feature" is consuming 6 engineers, is behind schedule and turns out to be significantly more complicated than anticipated and won't hit the end of year timeline. We don't know what uptake is going to be like. The resiliency feature could be shipped in 2 months and cut human interventions >70% of the time. Customers are already lined up waiting for it.
One argument is that the AI "feature" is forward looking and can bring more customers into the platform. Agreed. And then what? They are also stuck on the resiliency feature right?
I suppose there's the "better type of horse" issue. Maybe the engineering side can come up with something that users wouldn't think to ask for, or whose benefits aren't obvious until it's actually up and running.
> Being product-led means caring about your users. A platform teams should be product-led, not engineering-led in that very narrow meaning.
And where do product people get ideas for new features? If you're Microsoft you look at what successful competitors are doing. If you're anyone else you look at customer pain points, exactly as described in the article.
>It's precisely because of this framing and mentality that platform teams don't actually serve people well and are usually highly dysfunctional towers of people inventing work.
Spot on, I've worked in multiple small, medium and large companies. And these internal platforms almost always turn into ivory towers of useless churn.
Even if being forced to use it internally, in a lot of cases the projects ended up buying the same solution (or a working version of it) from an outside vendor when it came to fulfill actual customers needs on the outside.
If that wasn't possible we built what we needed ourselves.
If you have internal customers, I still believe you need internal incentive alignment. Otherwise what the platform builds and what is actually used and needed drift apart.
And you need at least one real customer project to dogfood the platform initially. Because most platforms start out way too generic with pure architecture astronauts [0] at the helm.
If you don't have a project-0 to test those assumptions against, you don't need to start that platform.
> And you need at least one real customer project to dogfood the platform initially. Because most platforms start out way too generic with pure architecture astronauts [0] at the helm.
The Rule of Three in Refactoring applies to this scale, too. If you are building a "platform" for one customer project, that's a one-off and quite possibly YAGNI. If you are building a "platform" for two customer projects, that's likely coincidence and still possibly doesn't show company-wide need. If you can build for at least three customer projects, that is a pattern, that provides real guidance on exactly how generic things need to be and better ideas how to abstract it.
Yeah, 1 is always better than 0, just that reminder that real "product focus" happens when you have multiple customers, not just doing one-offs for one or two customers.
I have seen it, over and over again, in relatively small orgs (under 500 devs) where the wrong kind of people are handed the job of platform lead. Project after project people don't want to use. The platform jobs in those places were handled by technical skill, but it was attached to disinterest of how others work. So even when they were sent in the general direction of a useful problem to solve, things rarely pan out, because they don't like to talk to their users. Therefore they write the tools they would want to use and are interesting to build instead.
Empathy for other developers, trying to have a mental model of what annoys them about work, and a general product centric mindset are the real requirements, but few people that decide how to staff the team k ow how to measure those skills. And besides, when people have them, they are often sent to talk to the actual customer, not internal customers.
i think that speaks more to the politics of “handed the job of platform lead.”
i’m many orgs the leads of any team are often people that are primarily interested in linkedin bullet points and being gently fondled and groomed by upper management.
this happens in product and dev teams as well. consistently.
I worked at a big company where the actual platform teams were serving the internal users ok, but then there was some middle layer of org-specific platforms that strangled entire projects. Eg some kind of custom database someone invented.
My efforts were often focused on cutting stuff like that out and using the company platforms directly, which was hard because there were reasons people had middle layers. The company platforms weren't bad per se, but they were in-house and hard to find expertise on.
You'd be surprised how VPs love to insert PMs. A famous game platform company, for instance, had two PMs for their storage team, one PM for their data team, one PM for the compute team, one PM for their dev tools team, if I remember correctly (the numbers could be larger, but won't be smaller)
> It's precisely because of this framing and mentality that platform teams don't actually serve people
I agree. When I was on the platform team (despite it not being called that at the time) our thinking was that feature developers were our customers. And our production was the tools they used. This meant not necessarily aligning production being only what was in production for customers of the company, but also in any service we provided to developers. This mentality had to be instilled into the team, and we had to remind people of this frequently, but it worked well.
Show me a platform team with a full product setup and I'll show you a platform team with over capacity.
Not to mention the average level of PMs will take a team like this down if you can even find one who can understand the type of work that the team is doing.
Yes. Which is why I don't really understand how any of it needs inventing. I'd rather read about how to find out what those teams actually need; and also how to tell when the answer is to build nothing new at all.
by "inventing" they mean that no one higher up in the org is going to tell you what your team needs to be working on, you need to do user research and discover the issues for yourself, and then come up with the right set of features that will solve those issues.
The technology is an agent you name. You give it a face. Maybe tune the personality. And, it only gets useful as you grant it more and more access to your accounts.
The technology is produced by a public company. And, in this case, the company is useful context for this story since it has just recently agreed to pay about $18 billion to resolve a multistate lawsuit claiming it intentionally designed addictive platforms that harmed young people's mental health.
You cannot detach the technology from the company. It's definitely not a neutral artifact.
I agree with the sentiment of the post, and Github deserves every bit of it. Tried codeberg (which uses forgejo) and I am very happy with it personally.
Sal could not get to his cousin's house every afternoon to do math with her, so he recorded the sessions. She said she preferred the recordings, because she could rewind them and because he wasn't sitting there watching her not get it. He put them on the internet. Then he spent about fifteen years building an exercise engine, a "mastery" model, and a teacher dashboard around them, and told everyone who would listen that the point was to move the lecture out of the classroom so the classroom could be used for something better.
And the essay's premise is: this man wants your child to watch a video.
How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.
Exactly, also it lacks a lot of context as to why they did that.
Possible (probable?) scenario:
- Marketing: "we found and fixed lots of bugs thanks to AI"
- Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually
Huh? Maybe a third scenario is that AI helped fix the bugs? Like described in the actual post you are replying to? This level of conspiracy theory is getting a bit ridiculous.
I think you’re right, my base assumption is that the models can code and can fix bugs, and can code more in parallel and faster than humans at a lower cost.
If Google are tackling lower value bugs with AI the number is in a way inflated compared to some utility measure (fixing a smaller number of worse bugs could be preferable) but it’s still things fixed.
There's no reasoning with the folks who are anti-ai and claim LLM's cannot produce anything worthwhile. They are basically flat-earthers or anti-vaxxers at this point. There is literally no amount of evidence that will convince them. They will always say that LLMs cannot produce anything but garbage just like the anti-vaxxers who state vaccines just cause autism. They are both fucking morons who no one with any grip on reality should indulge.
At my company there's a lot of discussion about AI and complaints that people run out of tokens within a day, but zero results are shown. No measurable (or measured) gains. Or nothing that people are willing to talk about, in any case.
At my company the performance improvements channels has been exploding, with people claiming giant improvements in latency, throughput, and decreased cost of the services. The cost decreases itself is order of magnitude more (at annualized run rate) more than we pay for tokens. YMMV.
I definitely make use of AI but in my experience I almost always could have done it better myself, the places where I threw AI at the problem I didn't care about the results being good, only good enough.
When we see memory and compute requirements for version x+1 of software decrease instead of increase I will happily say AI is the oracle people proclaim it to be.
In $COMPANY, for the mid-yearly review, the employees were asked whether their AI usage was 1/ efficient, 2/ adoptive (integrated in the way they work) or 3/ transformative.
Saying “never used it” or “I tried and it was useless” was literally not possible.
You'd have to be a pretty stubborn software engineer to not find AI useful for anything at this point, unless they're making you use Github Copilot or something far behind the frontier. You don't have repetitive tests to write? You never need to write a script to run something?
If you were the CEO of Amazon, would you be setting up channels for people to talk about their AI failures? The general way technology is deployed is that we try to find ways to make it work, because those are the most interesting. We're not as interested in all the ways it doesn't work.
From my perspective, some people are trying to use AI in the same way somebody might use a laptop to paddle a canoe. Sure, you can do it, but it's not a good idea. The fact that it doesn't work well is not particularly interesting.
If I steelman your position, I guess the ideal repository would be a set of cases where it's known to work well and a set of cases where it's known to not work well. A little bit like ProtonDB or SteamDB, perhaps.
Sure it is. It's just like the .com bust: people didn't know what the internet was good for, so they did a lot of stupid stuff with it before settling on the most interesting uses.
No, AI is more like: people do know what it is good for, but that's immaterial because hundreds of billions of dollars of debt were poured into it, such that the only glimmer of hope of getting an ROI out of it is force-feeding AI everywhere, good or bad, and normal people are sick of that.
That's fair. I still think there big similarities (people with money seeing it as the future and investing heavily, even if in reckless ways), but I agree there's a forcefulness to it this time that wasn't present before, for the reasons you outline.
> If you were the CEO of Amazon, would you be setting up channels for people to talk about their AI failures?
I would set up channels to talk about AI, encouraging both successes and failures, with proofs required for both, and punishing people who intentionally misreport on either.
That's fine, lies are more harmful than anecdotes are beneficial. There's no forum to discuss the best IDE and people still manage, in reality there's no need for a forum around AI.
Well, you're at Amazon. Is the Dev Improvement discussion group still around? I feel like that was a pretty good place to learn about neat stuff (in my case, Emacs-related). I enjoyed the discussion.
I'd say that one is not really an issue. In 2012 the Pinkie Pie exploit chain already required chaining 6 bugs to lead to an exploit [1]. Since then we've seen chains requiring more than 10 bugs (!).
If you fix any one of those bugs, the exploit is non-functional anymore. Sorry out of luck.
So if, say, for every ten bugs you fix, you introduce two new ones then it's still a very net win. Unless of course it introduces a bug so bad it becomes a simple exploit not requiring a long chain of exploits.
But in the case of browsers we've only ever been moving to longer and longer chains of exploits required to pwn a browser.
A great many window of opportunities are closing for dark-side hackers / north korean intelligence etc.: there were probably exploit chains still open for exploitation in April that just got closed by Google.
If anything, besides the supply chains attacks in amateur-land, the world didn't stop working: projects (not just browsers but OSes too) are being hardened left and right.
Using AI to find potential bugs is an amazing use case and there really aren't many downsides.
> The post has counts for everything that went right and nothing for what could go wrong.
I'm not saying there aren't a few downsides but the benefits are just too good to ignore.
This is the thing the anti-ai zealots will never admit. Humans fucking suck at writing code. They talk about software development as if it's only ever performed by the top 1% of the top 1%. They never acknowledge that humans make mistakes. No. Humans create perfect code every fucking time while LLM's only produce slop. It's such a fucking mind-numbingly stupid position that I have to think they have never actually worked in an organization which produced code as a value. These fucking morons want to pretend a human never introduces a memory leak when it's plain as fucking day that vulnerabilities that these "expert programmers" introduce to software are rampant. But no. The anti-ai zealot likes to pretend that only LLMs ever produce bad code. The only bugs in existence are due to LLM slop and not the literal decades of fucking slop produced by humans without any LLM assistance. It's so fucking tired at this point. They will never be able to admit that current day LLMs are far beyond the median human programmer. Their opinions are fucking useless at this point. These idiots think The Daily WTF started with LLM code. They are basically anti-vaxxers wrt to their grasp on reality. No one should take them seriously to any degree.
Or people don't ask questions for which the answer is known to be "None, really."
I get that many don't like what LLMs are doing to the industry, but this is just incorrect reaction to a very specific benefit that's proven beyond doubt (Security hardening).
Accept it imo - LLMs are solving very large problems that have plagued software security.
Sorry it is your reaction that is really weird. These are legitimate questions, and I have definitely seen Claude finding the wrong cause and then implementing completely incorrect/irrelevant fixes, only to find that it didn't work and need to start over. Not saying humans don't do the same thing, but LLMs are far from perfect, and it would be delusional to only talk about successes.
I've done security work before, and I've done it now with frontier. I've seen the difference first hand, and as the OP and so many other articles show, so have the leading experts in the world.
So either they are lying, or you may not yet be seeing and experiencing what they are. If that's wierd, ok.
You did not address my points or those in OP but kept repeating yours (that are not even relevant) in a handwavy way.
And that's fine. You can continue to live in your bubble. Others simply have different experience, and your asserting "AI is perfect" on a forum is not going to matter in terms of everyone's own, first hand experience.
