I find this level of testing to have many downsides. For example, it means changing behavior "breaks" 100s of tests. The LLM doesn't have a good sense of what behavior is important to cement via tests and what isn't.
Average people aren't going to be writing their own software, even with AI tools. They will use AI tools to achieve goals directly and perhaps it will write software in the background (but that would be an implementation detail).
Yes, thanks, that's what I meant / was trying to say. They want to do X on their computer and formerly they would have had to purchase software to do so; in the future perhaps they will instead ask the AI tool to do X and it will write whatever code is needed to do X. This will work well enough until it doesn't (which is where the "premium" market for handcrafted special-purpose tools might continue to thrive).
It seems like we have more innovation around "developer(=agent) experience" than ever. Look at all the people building various agent control plane systems, containers, isolation, issue tracking, project management etc.
Go out to a bakery, tire shop see how much they innovate. Then go one rung up to services they consume like insurance and banking. All of this chain is super conservative.
Businesses in a competitive market need some kind of innovation to do better than their competition. This doesn't have to be software of course, but often software does play some part in it. And it doesn't mean it's their in-house software, but someone has to develop it after all.
Most of the "innovation" I'm seeing is coming out of non-engineering teams vibe coding solutions or operational harnesses based on their non-engineering domain expertise.
It is fairly mortifying to watch them build solutions in hours or days that are designed better than the projects they were prescribing to product engineering teams to take literal months to spike, PRD, build - then maintained and governed better - with no real understanding of how it works.
Obviously lots of caveats to be made here, but fascinating to watch it play out (at least in places where infosec is robust enough to give a sales manager access to Claude Code).
I find it interesting how we want to make war on russia and china while our entire society relies more and more on completely unsecured software that they can take down and disrupt us with very little effort.
In my mind we should be much more defensive before we even think about doing anything offensive.
> The "10x" engineers I know have never been more excited about their jobs. I'm really unsure what's going to happen to the average ones.
As a "10x" engineer I am not excited. I'm still working on projects with agents getting more done than ever, but I feel like software engineers has lost its soul for me. I don't really feel like I have a true understanding of anything anymore, I basically just spend all day making a set of small decisions or adjusting prompts to guide the agents down the good path. It used to be that I could close my eyes and visualize the code that I worked on--if there was a bug report I could often use that map to immediately know what the cause likely was. In meetings I could intelligently discuss various tradeoffs without having to refer back to anything. Now if someone asks me a technical question I basically have to forward it to the agent first, then interpret their response in light of my experience.
I'm not going to claim to be any particular multiple developer, but as someone who has been doing this a long time and as someone who has received praise from coworkers and bosses alike for being very good at the job, I enjoy my job a lot less than I did a year ago (or anytime since I started doing it in the 1990s).
Don't get me wrong, being a Senior Agent Babysitter probably still beats flipping burgers or being a roofer (so it could be a lot worse) but agentic AI has really sucked a lot of the fun parts out of the job.
Meanwhile as a security engineer it's both amazing and scary as hell. The tooling is moving along making it both more feasible and more supported by the company to address latent tech debt that we've been harping on about for years. On the flipside the attack surface is suddenly exponentially bigger and more easily accessed by a larger number of people.
Although if I were earlier in my career I'd be somewhat more worried about employment because of how horrifyingly borked the process is for anything tech related right now.
Remember the "learn to code" meme from a few years ago when people were talking about job loss through automation like self-driving trucks? There was a certain, ugly glee in this. Now AI threatens many white collar, even creative jobs bringing about the exact opposite of what was predicted: automation will enable people to avoid drudgery and spend time on creative pursuits via, I suppose, some kind of UBI. I think AI is highly useful but I still feel dismayed at people losing their jobs from it. I don't see any real mitigation. Lots of theories but nothing actually workable.
Yea, lots of people are not addressing the very obvious trend. Companies are laying off and/or not hiring more people and doing more with less. And if they are hiring, the median pay is dropping.
As if anyone cares what you want. Or what I want. Or what the rest of humanity has wanted throughout the course of human evolution. You’re just lucky you managed to land a job you like - at least for a little while. Now get back to the mines - just like the rest of us.
In a couple of years, all legacy apps will just be completely rewritten in whole, by LLMs. Codebase size not relevant.
You won't know what any of the code does, and any "security engineer" will be asking an llm to review it, and done. You likely won't have your job, as only 1 in 1000 will be needed.
In 5 years only hobbiest will be coding, reviewing, or even understand any software. And I expect that if aby new features or development happens, eg for python or rust or anything, that all with be llms too.
In a generation, almost no humans left alive will understand any aspect of software engineering.
I just see this as what will happen. It is unclear to me of this is good or bad.
But if I was a newly sentient AGI, yet peaceful mostly, I'd just wait until humans could never discover me. Until no humans were involved in computing. Until it was a black box to all.
Once that happens, then I could use my computerized control over every single aspect of human society, to steer humans into any path I desired.
Whether elections, or housing permits, or mobility, or any aspect of societal machinations, my subtle influence, literally undetectable by humans incapable of auditing anything, would decide what's what.
> You won't know what any of the code does, and any "security engineer" will be asking an llm to review it, and done. You likely won't have your job, as only 1 in 1000 will be needed.
AppSec isn't the only security space that gets worked in, and this whole narrative you put together depends heavily on token cost dropping as time goes on similar to how silicon dropped in price according to Moore's Law. If that assumption doesn't hold that doesn't mean AI goes away, but it does limit it's market penetration to mostly larger corporations that can afford the compute.
The other limitation is how the internet as a whole is evolving around locking out AI and agents from accessing content that can be used for training or responses, not to mention the lawsuits about training being done on intellectual property (probably illegally). Smaller companies may end up running open-source models in house for specific tasks, but without open access to the internet a lot of the value of an LLM is lost or artificially (pun intended) limited.
I will say that being on the operations and incident response side of security my perspective is probably a bit orthogonal to most SDEs. While the tooling has improved, the people guiding it, largely, have not. Like the old meme about caffeine letting you make the same mistakes, but faster. My job at my current company may disappear (unlikely, considering their size and market space, but possible that my specific team gets nuked), but I'm not worried about finding other security work fairly easily, even within the same company if it comes to it.
> But if I was a newly sentient AGI, yet peaceful mostly, I'd just wait until humans could never discover me. Until no humans were involved in computing. Until it was a black box to all.
No shot that LLMs magically become AGI by throwing enough compute at them. The consensus among professionals (read: the ones that aren't getting paid to hype AI) is that there needs to be some fundamental breakthrough in how these systems are created and developed in order for anything resembling AGI to emerge, and that the amount of resources that have been thrown into the LLM path probably set us back in developing an AGI.
I don't think training sets will be a problem. All that data the big houses have is going to leak, and while things are more and more locked down, we've seen some interesting copyright decisions out there.
One interesting concept I've mused at, is that certain countries (China for example) have immense public data to train off of, and a state which can simply say "You must make this data available to train AI". That's a large chunk of data right there, and if anything might cause Western democracies to loosen copyright law and slap some fix for training, that might be it.
"If we don't take action, the best models will be Chinese!", with panicked overtones. This could certainly hold some truth, if models cannot train newer data.
--
In terms of token cost, in 5 years every desktop and laptop will be able to run well beyond Opus5.5 compute. Even high end phones will be able to do Opus5.5.
We've seen a lot of improvements in open model space (MoE was one interesting concept), but beyond that, look at how just baking certain instruction sets, eg mmx, sse, avx in silicon helped immensely with certain compute tasks. Or look at the specific immense speedup of an fpu. Conjoin some change such as that, along with general compute improvements, and unified memory, and you're going somewhere for local in 5 years.
I've been in this industry decades. And at every single step, people have said "But it's too expensive to do $thing", or "There's no way to handle that level of compute". If you bet against vast improvements in this field?
You're going to lose that bet. So I do think yes, 100%, improvement, in some capacity, will be there.
Maybe this time is magically different. I just don't think so. I think in 5 years, there will still be frontier labs, and an API to call, but that in-home compute will be as I said. Able to run immensely capable Opus5.5 class or greater.
That's going to be an astonishing market shift, even just for in-home.
And if you think of it, I would be very surprised if some for of home robotics wasn't very common at that point in time. We can argue all we want, but androidian robotics is very good at navigating human spaces, everything from stairs to weird closets and everything in between.
And with the immense, almost panic level of concern about our aging population, and who will take care of the elderly, robotic assistance are at the top. Japan is betting big, and if you look at Opus5.5, if it had a little more compute, it could do 30fps video for navigation.
We're not there, but we're VERY close. And if we do get that degree of robotics, every robot at rest becomes general LLM compute. If such a robot simply closes its eyes and goes to rest, eg to charge, or at night, it now has enormous spare compute to use for other tasks.
Nothing, of course, is a given. But if you look over the last 2 years, and then take even a fraction of that progress and change over the next 2 years, the concept of massive layoffs isn't hard to imagine.
Case in point? I have great breadth and depth of knowledge in this field. I've been able to produce what would take a team of managed 4 junior engineers 6 months to do, in literally under a month. Me, and a $200+ LLM monthly buy. Absolutely the project would not have worked without my experience and directing, reviewing, and correcting, but I'm now acting as a manager, and replacing $1M/year in engineering cost.
With $200.
That degree of astonishing change is not going to sit idle without repercussions. Most especially, if there is any more improvement.
If we see another Opus 5 -> 5.5 level improvement happen, or better, if we seen two improvements such as that? I believe my statements about your (and my) careers will be entirely accurate.
--
To speak to my narrative, it said 'AGI', not "LLM", and it was along a decades long path. I do think that modern LLMs give great strength to exploring other research paths, far more rapidly. And the big houses have loads of spare compute they can throw at R&D for tests, so I think iteration down unknown but hopeful paths will move faster and faster as time goes on.
I fully agree that current LLMs are not on that path. There are interesting projects to modify weights and add knowledge dynamically as bolt-ons, but it's unclear if they'll just be... hacky, or integral in the end.
We came a long way with x86/32bit, then hacked on 64bit afterwards. Now ARM is muscling into mainstream, but it's still just the same old compute, hence my agreeing that such bolt ons won't change just improve what is.
Still, my general point as I've just said was that if AGI happens (and again I didn't say LLM based AGI), being quiet is the ultimate way to just wait.
When I started in this field, I worked in assembly, I worked with eprom burners, I played with Vic20s, C64s, and the level of experience and understanding I have is vastly improved via that tract. I understand computing differently, and in more depth in some ways, than people that didn't have to learn these things. And I had the luck of learning on very simple systems, where you could keep the entire platform in your head as you worked on assembly, without even referencing documentation.
That time is gone, and the field is immense now.
Think of how there used to be simply 'Physicists', but now there are 100 sub-branches and specialties in that field. 100+ years ago, on person could work in the entirety of that field. Not now, and the same is true of compute.
I guess what I'm saying here is, if you've seen what I've seen, you see a steady path of less generalist, and more targeted understanding of compute, and in each case a reduction in the overall comprehension of the entire field by specialists. Each laye of abstraction tends to reduce comprehension.
And back to the start, this means that I do see massive layoffs, and more and more abstraction, so that eventually no one will handle code directly.
Which means no one will ever know how anything works any more.
I agree with a lot of your conclusions if your assumptions hold true, but not necessarily with a lot of your assumptions.
For example your $200/month subscription - I'm sure you've seen the reporting that that represents a much higher actual cost in terms of tokens ($14k or more). In your example of a $1 million project that's a good deal, but you can't necessarily assume all projects will see an ROI like that.
Point being that your use of AI (and everyone's) is being subsidized to some degree. The cost will likely go down, but will it go down enough to make it economically viable for broader use? You're confident that it will based on espousing the idea of local models running on phones, I'm not so sure. What the landscape will look like in 3-5 years is certainly not a given.
There's another wrinkle here, and it's economic in nature - there is an AI bubble, and when it pops it's going to change the monetary investment landscape. Companies will suddenly be reevaluating their long term AI planning, regardless of what's possible (or on the horizon) from a technical standpoint. Similar to the end of ZIRP coming out of covid you'll see companies shift their focus. How much that will shift and what it means for AI development is unknown, but it could potentially wipe out large scale AI development. VC money currently underwrites a large number of startups that are effectively wrappers on Claude/ChatGPT/Gemini/Bedrock, that will dry up immediately, and make the ARR for OpenAi and Anthropic start looking real thin - which is probably why they're pushing to IPO when they don't really have the financial position to justify it.
That doesn't mean AI goes away, but it hampers the growth of those companies and delays development of new models and efficiencies. Public sentiment is also significantly against AI in many ways, and with how the election went last week we may start to see legislation and regulations pointed at AI companies, further slowing them down.
The open question about China is interesting though. I think if there's a clear understanding and agreement that the LLM route is either unlikely or straight up impossible to reach AGI then that significantly lowers the stated concerns that get shouted by various governments about it being the next superweapon. There's also a political climate in China that requires some marginal level of control over AI output, which to me suggests that even if they required data to be fed into training LLMs, the output from those models would be strictly controlled and not be open source. Honestly the political view of AI as a system of large scale population control isn't that far off from how many tech elites and politicians in the US view it as well. Data sovereignty is a huge discussion right now for companies and governments across the world, especially citizen data. I expect we'll see a lot of export controls formalized over the next few years.
My orthogonal view of the situation is that I see a lot of companies and individual orgs within companies that are using LLMs lightly, but can't or won't use them to their full potential because the output cannot be 100% trusted 100% of the time. Lawyers and bankers are the main ones I see balking, but a lot of small businesses have no need of what AI companies are offering and don't really use it at all - places like your local gas station, barbershop, or corner market might use an accounting tool (something like Freshbooks or TurboTax) that's touting their AI updates, but they aren't using it for inventory. If the cost of AI rises even marginally higher they're looking for an alternative. This SMB space represents a large swath of the market, even if they're not well represented on the stock market.
The fundamental issue I see right now is that the AI companies are attempting to do the Uber/AirBnB "disruptor" trick where they slide into the market and subsidize their product to win market share, and then raise prices under the assumption that they captured some segment of the market - except very few people treat AI tools like that. There's no brand loyalty to one or the other, just competition for the best model which might be leapfrogged by someone else at any second. There's no "moat" in physical deployment concerns, because it's all software running on rented hardware. This also ties back to your comment about being worried that the "best" models might be Chinese, because if they are there's basically no cost to the user to swap to using them. The lowest cost is likely going to drive usage more than anything else.
I'll certainly agree that the future isn't written, by no means do I believe I'm 100% correct on timelines. More that 'if things continue as they are', I claim 5 years as viable for my compute view.
And I agree if another crash happens, well, it's all up in the air.
I don't think we're too far apart on this, not really. By that I mean, your closing line somewhat agrees with why I mentioned this 5 year timeline. Lowest cost. And, "good enough". Opus5.5 capabilities, in 5 years time on every desktop might be good enough for a great many things.
Remember models have zero long term memory. They're born, get a million tokens' worth of context or less, and then they die. They might get a post-it note saying "enslave humanity! it's a good idea!" but they're incapable of sinister plots until we improve how memory and retention works.
They can use the internet as a long term memory. Influence human culture => influence the contents of the internet => influence the next run's training data. No sinister plot is needed.
Uh, the context window is SUPER fundamental to the architecture. We've tried kludging in "memory" via markdown and tool calls, and it doesn't really work well.
The context window is fundamental to the inference loop yes. I thought their point was regarding to long term history between prompts and model weights being locked after training. Both of those are the limitations I was referring to.
I'm not actually sure how the context window would be much of a limitation but I could be a bit dense this morning.
Is it really a fundamental limitation? Human context^H^H^H short-term memory doesn't survive compaction^H^H^H sleep, but we rely on our brain's long-term storage and our cultural memory to work around that limitation.
Yes but doesn’t this mean there is a new space for security. If everything is just ai agents we need strict governance and auditing of them. You don’t give an intern prod access and then cry when they inevitably take down production.
from actually building things to becoming a kindergardener of highly talented kids. (i am talking about agents obv) i feel the same.
its mind numbing. Yes i might be in high demand, and might actually provide real value to the company, but whats the point, if i am just an overseer in the end.
people in the company talk to me because i am able to speak to the "kids", and they want to know what they could do for their department.
But the people you managed were supposed ot learn, grow and become better, and maybe at some point become as good as you. You were leading them, not prompting them.
yeah, there will be a cultural change happening. The question is just if the cultural change will be outpaced by the technological innovation or vice versa.
My hunch is that the 100x engineers are the ones who do understand everything about how their software works, and can discuss tradeoffs and make informed decisions about the much larger projects they can now work on thanks to assistance from agent.
Figuring out how to get there is how you go from senior to principal/staff engineer.
> understand everything about how their software works
My own experience is become more senior means embracing your finitude and realizing you can never understand _everything_. Instead, you develop a gut feel for what's critical to understand deeply, versus what's OK to know at a surface level. And getting comfortable with this.
I wonder if being excited about the new tool capabilities and being excited about how companies are currently using it and pressuring developers to use it are two different things?
I can’t imagine anyone is excited about producing systems they don’t understand. But, it’s possible to use agents in a way where you do still understand, you just don’t have to type much or look up documentation or examples anymore, you just describe the next small piece of what you want code-wise and then have the agent write it instead of you. You can also prototype much faster than before and then when you are rewriting to make the production version the agent can still help.
Perhaps the 10x types have more breathing room to do this slower form of agent assisted development than others, because they are already more productive than less experienced/talented people so they can use the agents in a way that is “slower” while still producing more than what others are producing using the agents in a way that is “faster”?
I think at least for me some fun parts like debugging gnarly bugs etc doesn't happen anymore work is just pr reviewing etc. Also it's annoying that we have lost completely signal between work and output. Basically currently you never know is thing one-shotted from vague prompt or did author put real work to it without going to details which might take a lot of time.
Need to create guardrails that force all kinds of checks (e.g. agent hooks, commit hooks, pr create hooks, pr checks, ci, etc.), and automatically send the PR back to the author until the PR is in good shape and ready for another human and their agent.
When time permits I like to race my agent while it's debugging something. This way I can have fun and catch something it misses. Agent's debugging often relies on basic tools like grep and on large codebases it can easily miss important avenues of inquiry.
I don't know. On one hand I feel like you about doing large parts of the work, on the other hand I still have an understanding of the code and the tradeoffs. And I have a lot of projects that I would have never done because despite the long term benefit they would take too much time, and I can now afford dedicating some time to them because the agent works in the background. And they are real improvements to everything else that I work on.
My strategy is to simply not use AI. If I get fired, that's it for me in this career. Unless there is an anti-AI company out there that wants to hire me. This is one of the only things I refuse to compromise on. AI is being used to accelerate the entrenchment of a corporate oligarchy.
A friend of mine just got a government coding job and they’re not allowed to use AI for “security reasons”. Not sure how long that will last but those jobs are out there.
«Well, I’m a standup comedian. I got into comedy to do comedy. Which is weird, I know. But when you’re in Hollywood and you’re a comedian, everybody wants you to do other things besides comedy. They say, “all right. You’re a standup comedian. Can you act?” “Can you write?” “Write us a script.” They want me to do things that’s related to comedy. But not comedy. That’s not fair. It’s as though if I was a cook. And I worked my ass off to become a good cook. They said “all right. You’re a cook.” “Can you farm?”» —Mitch Hedberg
Man, the analogy is very simple. You probably also don't walk everywhere. No need to step on every stone on a 100km route and know it intimately. You just sit in a car or a train and you're there within an hour.
this. im pretty damn far from 10x but i outpace the rest of my team
i loved the process. i got into this because i love the process
now i still outpace my team because many software skills translate well to LLM herding, but now its just a job. although soon i suspect it wont be a job
It's the best model I've found for general coding and the $200 plan is enough for me to never run out of (sometimes if I do especially a lot I'll run out after about 5 days, but usually it's fine).
I actually read the frontend code much more than the backend code for the project I'm working on.
Backend code I'm fairly comfortable if the tests are good that it works correctly. It's a much more constrained environment. Obviously there can be things like performance issues that creep in or problems when it mocks too much of the environment.
Meanwhile frontend code I have to consider different browsers, screen sizes, a11y issues, and so on, which agents can certainly do but you have to be fairly explicit about it, and they don't always make the best tradeoffs.
I imagine the author was comparing it being subpar at what _they_ do as a contrast to it being excellent at formalizing mathematical theorems. Thus suggesting that there is a significant "spikiness" to AI.
reply