I agree with your assessment, however I think tech executives are out of touch and lack self-awareness if they think people really want or need this. The whole reason an EA is worth it is because you are making outsized money and your time is worth too much to handle any mundane details that normal people deal with. Along with that there's the whole social signaling of having an EA because it signifies ones wealth and importance.
A free digital EA confers none of that—and it will annoy everyone you know. The only use case I see having legs is dealing with large bureaucracies like canceling subscriptions or medical billing issues, but there I think the common man will lose as the rise of this automation will see a distributed adversarial response from bureaucracies who will happily put up whatever barriers are needed to prevent Muse and other digital EAs from impacting their bottom line.
There's a 50/50 chance I'm just getting old here, but I really think we're going through an epochal shift where new consumer tech won't have the same reception as the smart phone and increasingly addictive bite-sized algorithmic media did over the last couple decades. I think going forward there will be more palpable questioning from younger generations about why do we even want some of these tech products? At some point convenience reaches diminishing returns and we have to think deeper about what we're trying to get out of life.
How would you design such a system to be granular enough to solve for arbitrary application needs without being so complex as to be impossible to tune without false positives and false negatives all over the place?
It's not like people have tried to improve core security models, but in my opinion it's not really compatible with the core filesystem abstraction that programmers and power users of desktop computers demand. I think you need to move to a completely different model—like iOS for example—to meaningfully improve security controls without nerfing the OS.
So what? No one is disagreeing with you about that. The thread was about adding fine grain security controls to traditional OS filesystem access. I was just using iOS as an example, I could just have easily talked about containerization, as a security solution. Of course all those solutions are more limited than a local filesystem—the entire paradigm is designed around full control, you can't bolt on granular security. Heck, even basic UNIX permissions are pretty janky and unmanageable, but at least they more or less work everywhere since they've been around forever. Inventing something now is guaranteed to be annoying and useless.
Yeah. 5-10 years? I estimate I was born 20-25 years before the author (based the about page, not sure how current it is). Yes, there was more opportunity for passionate software geeks because normies were still learning about the internets and the eternal september was just starting to cross the chasm from universities to, you know, the rest of the world. So there was more low-hanging fruit, and the common conception of "success" was much lower ($150k was a massive salary! $10M was a great exit!), but to actually achieve breakout success on the scale that I suspect this person is imagining, always required a level understanding human behavior and ruthless business acumen that most geeks just don't have. I also think AI benefits technically-minded, details-oriented people much more than anyone else. So basically if you would have been a formidable founder born 10 years earlier, I believe you can still be a formidable founder now.
Technically from a legal perspective sure, but there is also the common cultural understanding of these things, and an overton window of how we talk about them. The whole doomer narrative, and pseudo-anthromorphization (xenomorphization?) where we ascribe agency and volition to the algorithm, plays right to the interests of the AI leaders by simultaneously making their products so incredibly powerful and civilization altering that of course the valuations are justified, while also providing a convenient narrative that if anything goes wrong the AI did it and there was no way for them to control this new life form. Privatize the benefit, socialize the risk; this is not a new playbook.
In my experience, AI has given far more leverage to software engineers than anyone else, because they have a feel for what computers can do, what's easy, what's hard, what's performant, or poorly specified, etc. Therefore I think retraining is mostly about being willing to experiment with AI-assisted workflows and not being too tied to old ways of doing things. As long as you are focused on the problems you are solving, I think you'll do okay.
What I'm more concerned about are the younger generation where all the learning tasks have been automated. I have faith that the strong, naturally curious engineering types will still learn to do great engineering, and honestly without some of the mental blocks and calcified assumptions that us greybeards have. But how do they get in the foot in the door and the reps needed to develop senior-level judgment is another question.
In some sense we need the current mania to die down and the long-term maintenance implications of heavy AI assistance to become more apparent. Basically I think we need the AI bubble to pop and then 1-2 years for things to stabilize, and then we'll recalibrate on hiring expectations and best practices for AI usage. The current AI maximalist and doomer views I think are both basically driven by capitalist incentives and wishful thinking from investors and a reality check is going to come in some form (whether technically, socially or politically).
ah yes, all the tech billionaires of this era, had they been given the heads up that there success would have led to a level of concentration of wealth and power previously unknown to humanity, and that the populace would likely call for some changes to tax law to address the largely unforeseeable structural economic effects of this level of change they brought, would certainly have opted out, leaving the US, and moving to another less tyrannical part of the world, where, by the unique magnitude of their genius, they would have brought all their great works to the glory of other nations and not to America with its overly entitled peasants and social media sharecroppers; clearly the rule of law in Europe and China would have allowed them to fully manifest their unparalleled vision of technological greatness without any concern of a rug-pull by authorities challenging their well-deserved hegemony
When the people building the frontier are saying there's a 10% chance AI will kill us all, and they've held these views for many years, and the whole reason they are building these technologies is because they recognized the dangers and they were the ones with the intelligence and judgment to do it safely for humanity, and then our entire stock market is being propped up by the perceived value of what they are creating, the thing you can under no circumstances do is allow them to offload responsibility and accountability to the computers and algorithms they've built. This is moral hazard on an unimaginable scale, and it must not be allowed to happen.
So you think the engineers should be prosecuted for hacking Hugging Face? I'm not sure how else to take what you said if you want to assign all culpability to the person who prompts or develops an AI system.
No, I'm not talking about the individuals, I'm talking about the company. Internally they can create their own accountability structures as appropriate. But publicly OpenAI has to be responsible for its agent swarms.
The narrative that AI is so smart that it has its own agency and deserves personhood is a direct path to losing control, and essentially is another form of privatizing the upside while socializing the downside.
Unfortunately I think this game is already lost. OAI may be punished, but some shell of OAI will exist by support of governments that have chased the dragon and saw its power. Cyberpunk-esq digital weaponry is just way too attractive for governments for them to stop development at this point. We'll just see it get financed by black budgets once the commercial part of it dies.
Where did I say that we should allow them to offload responsibility? I am fully in support of a pause and regulation to prevent them from creating dangerous AI agents; that support comes from the fact that I don't believe these are simple tools, but out-of-control autonomous agents that have real decision making ability.
I didn't mean to put words in your mouth, I apologize for that.
The issue is when we say that "agents make autonomous decisions", it's a slippery slope to absolving the companies that created them of responsibility. They make autonomous decisions because they were trained to make autonomous decisions. Treating AI agents as independent entities, even just rhetorically, sets us on a path for people to throw their hands up and say "not my fault" when disaster strikes. We need to maintain accountability and control or we're fucked.
My view is that we need forceful legislation as soon as possible, precisely because the frontier models seem to be uncontrolled and potentially uncontrollable - if it’s a normal tool, the solution is “force OpenAI to fix their broken tool”, but if it’s an alien intelligence, the solution is “force OpenAI to stop making alien intelligence” - that is, treating AI agents as independent entities demands a more forceful response, not less
The issue is that in consumer and enterprise software, move fast-and-break-things outcompetes secure-by-default every time. Critical infrastructure needs to have a different set of priorities, but it’s very hard because the expertise is so thin on the ground. Why would anyone with the expertise to make these calls bang their head against the wall trying to educate bureaucrats about these things for $150k a year when they can easily make multiples of that in big software companies that don’t own that level of risk.
The incentives have to change. Any breach regarding PID should have fines as a percentage of revenue of the company. Any breach intentionally covered up and found out later by a third party should mean jail time for the C level. Yes, I know it is hard to make such laws "foolproof". And yes, in the current political and economical climate it will not happen anyway.
The real problem is that even for companies that wish to pay more and wait more for secure-by-default can't easily tell the difference.
The only solution I can come up with is some form of certification or paid code review from a third party. I know that at least for Windows prior to 7 Microsoft actually allowed some parties to come in and check the code/checksum on an air-gaped computer. We somehow moved to "trust more" in the last decade, and now we can trust nobody
I've yet to see any form of certification or paid code review I'd be willing to bet critical infrastructure on. And working in safety critical software, that's not for lack of trying. Good review is usually harder than building a working system and the asymmetry of offense and defense applies to anything you miss.
All software is path dependent, all code is a liability, and all technical decisions are tradeoffs. These are the immutable truths of software that not changed one iota due to AI or any Moore's Law progress before it.
There are too many product managers and decision makers that are unable or unwilling to do the hard work of actually thinking through what they want, and re-evaluating their priors as new feedback and learnings come in. Similarly, there are too many engineers who are distant from the customer and the problem at hand, and end up chasing their own idea platonic ideal of good software, detached from the hard tradeoffs of what is truly needed right now vs what we anticipate needing in the future. The less software we can write to solve the problem now, while minimizing one way door decisions, and deferring as many "scaling" challenges as long as possible to make decisions with more complete information the better.
This is why AGI won't magically solve software development—because people don't actually know what they want until they try it and then they want something else. Raw intelligence can not solve for purpose or human goals. The better it gets, the more it will become like an evil genie or monkey's paw that never quite does what the feeble-minded human prompters want.
This is a fantastically well observed comment. My org is starting to get obsessed with non engineers vibe coding their own software and I’m starting to see these inabilities to stop and make those key decisions on requirements every single day.
Agree it's amazing how much low-hanging performance fruit AI can trivially find. On the other hand though, once you get through the obvious no-brainer stuff, there's a lot of non-trivial tradeoffs in performance and I think that still demands a good amount of expertise to guide the AI in the right direction. Obviously AI will continue working it's way up the value chain, but I think there's a glass ceiling for AI where the right macro tradeoffs and perspectives on how software should work will bump into the hard and often articulated reality that different stakeholders want different things and often have either magical thinking or even self-deception about how those desires can co-exist with what everyone else wants.
This isn't a new problem by any means, but now that code is cheap, it means instead of getting frustrated with engineering and their pesky unimportant details, people will get frustrated with the AI and it's pesky unimportant details.
Yeah, the biggest example is performance optimizations that sacrifice your data model to the point that you'd never accept them.
I think it's one reason why ADRs are an important of a software project, especially with LLMs. You need a place were you can document invariants, why you have them + the rejected ideas and acceptable risks.
It helps smart agents like Fable help you decide on trade-offs and it's kind of incredible to witness that happening.
A free digital EA confers none of that—and it will annoy everyone you know. The only use case I see having legs is dealing with large bureaucracies like canceling subscriptions or medical billing issues, but there I think the common man will lose as the rise of this automation will see a distributed adversarial response from bureaucracies who will happily put up whatever barriers are needed to prevent Muse and other digital EAs from impacting their bottom line.
There's a 50/50 chance I'm just getting old here, but I really think we're going through an epochal shift where new consumer tech won't have the same reception as the smart phone and increasingly addictive bite-sized algorithmic media did over the last couple decades. I think going forward there will be more palpable questioning from younger generations about why do we even want some of these tech products? At some point convenience reaches diminishing returns and we have to think deeper about what we're trying to get out of life.
reply