Given development speed is way up, I'd be interested to see what a from-basic-principles simple and efficient OS and programming system might look like. Or people trying radically different versions of efficiency and simplicity, like an OS that uses an existing battle-tested sandboxed execution environment like v8 for userspace safety/isolation.
Fundamentally, many inefficiencies were justified for developer productivity, and I'm not sure that makes sense anymore, and the cost of larger projects with clear abstraction boundaries has plummetted.
>Fundamentally, many inefficiencies were justified for developer productivity, and I'm not sure that makes sense anymore
Having lived through this period, I would disagree, quite the opposite we tended to develop in C and assembly specifically for runtime efficiency because we had to.
For developer productivity you could opt for Lisp or SmallTalk or some such, which ran much slower until high quality compilers for these became available, and even then your target machine might only have a C compiler.
I read GP's comment differently, that we ended up where we are today (bloated languages, IDEs and runtimes) due to chasing developer productivity at all costs.
Now, developer productivity is becoming less important as developers are factored out entirely.
We will end up with something that is easier for AI to write, efficient to execute, and possible for people to audit.
Rust is becoming popular as LLM output. The resulting code is very efficient. The compiler helps keep the LLM on track and holds it to a minimum level of quality. Humans can read it if they want to. But it is not great to read.
Python is also a popular target. Easy to audit. LOTS of training data. The “one way to do it” philosophy of Python helps. But the resulting code is not very efficient.
I think we will end up with a new language that balances all these things better.
Rust isn't my personal favorite for having agents drive development because the compile time penalties add up fast unless you stay on the agents with prompts/AGENTS.md/etc directives to really keep it from re-running full compile/test suites every iteration.
It can be the difference in a loop that an agent can crank on in Go for 15-30min while I get a coffee vs taking multiple hours in Rust, sometimes.
Anything agent-focused language/ecosystem wise needs to focus on small-blast-radius/fast-iteration.
(Unless/until there's a generational leap in the raw coding ability of these tools to be able to get it "right" (for larger-and-larger definitions of "right") the first time.)
I don’t think we will. Well maybe for a bit. But I think eventually LLMs will produce a kind of byte-code or IL.
Maybe it will have human words inside of it to explain the parts.
Most of the way we organize code is to support our brains. Code reuse via classes and functions. Extensible patterns. Classes.
AI doesn’t need all that stuff. Consider, AI needs what assembly needs. And assembly doesn’t have any of those conveniences.
At some point AI will stop writing code for us to pretend we own and start writing it for itself. I am curious to see what conventions arise to support a programmer whose brain has infinite storage capacity and can figure out what existing code does without having to remember or read documentation.
Not all of that abstraction is solely to help us understand. Much of it is to make iterating on the code over time simpler and more reliable: avoiding unintended side effects, for example.
This was my intended meaning -- the clearest advocacy of this was from DHH about Ruby on Rails, although he's changed his mind about big questions lately and suspect he no longer endorses that set of trade-offs (I don't either, _sort of_ -- I think developer productivity means having simple ways to enable large-scale testing and oversight while minimizing context switching costs, although I think we are all figuring out what that means in practice).
This is the thing - when the cost of iterating and grinding compiler warnings drops to virtually 0, what new paradigms might we invent to exploit that? Rust gives us an inkling of an idea of what might be possible, but that still predates LLMs and thus still makes concession for human ergonomics. What are structures, guardrails, and patterns, that our OSs, frameworks, and toolchains can put in place, which would result in whole classes of problems going away completely, at the expense of massively increased (but mechanical, and easily automatable with a prompt) developer effort?
I imagine a kind of language which is just a bunch of declarative statements about conditions and what should happen. Basically BDD. Maybe we will just write the “unit tests” in human and AI writes more comprehensive specification to fill in the blank and for us to modify. And then AI makes code that satisfies the spec.
I don’t believe that source code is a spec. Source code has real time manipulation a kind of temporal dependency. Specs are declarative.
The difference is a spec would say: adding two and three should equal five. Code says you have a variable give it the value two. Now add three to it. Return that value. Maybe a unit test to assert it returned 5. Code defines how a spec is satisfied.
v8 is pretty complicated, and anything that depends on LLVM is going to be huge, so I think I would start smaller. You could try building around something like NuttX or an L4 + tcc or cproc and see what it takes to get to self-hosting as well as ability to do some job you care about. To me the bigger question is that last bit, you can shrink the platform code quite a lot relative to a typical Linux but what's the situation where that's a good trade vs say cross-compiling from a full C++ or Rust toolchain?
Ahhhhh this is a common misconception and it's dangerous. V8 is not a battle tested sandboxed execution environment, not unless by battle tested you mean battle wasted. V8 is full of sandbox escape vulnerabilities practically "by design", always has been, probably always will be - and the Chrome engineers knew this from the start which is why from day one they invested so heavily in kernel-level sandboxing.
But the kernel sandbox is NOT A PART OF V8. It's not even reusable or modular, it's a hard coded part of Chrome's startup sequence and tuned for Chrome's exact needs. If you embed V8 via its API you don't get it.
I've seen this misconception several times now including in quite serious places, where people talk as if V8 on its own has a robust sandbox. It doesn't! This will blow up in people's faces eventually.
If you want a JS engine that's secure by design then the only one I know is GraalJS, as it's written in a memory safe language and uses partial evaluation to create its compiler so you can't get interpreter/JIT mismatches. It also has a proper sandboxing API and can transparently run code out of process. They're experimenting with integrated Landlock support now, so it actually will get a built in kernel sandbox on Linux out of the box, unlike V8.
This issue vexes me enough that I wrote a whole article about it:
Can you point out where? Looking at the METR report, the only place I see discussion of humans being involved in the sequence of events is two short paragraphs on page 30 where a security investigation into the artifactory issues led to a pause before ExploitGym experiments were resumed. There's no deeper analysis on what was found during that investigation, nor why training was resumed even though the issues weren't mitigated. Another part discusses The agents choosing not to actively email a human researcher, but not the human researchers actively looking for evasion.
1. Failure to Care or Respond. The biggest holy shit moment, to me, remains that OpenAI on multiple occasions had teams that found out about the message board, knew that agents were in communication, and they disregarded this. The first known warning was in late May. The warning on June 27 was unambiguous.
“Responders investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot. At this time, the on-call response staff advised that stopping the evaluation run was not required.”
2. Failure to Monitor. The entire time, there was almost no monitoring of the situation, of what these AIs were up to. That’s how this was not caught.
3. Failures of Infrastructure. OpenAI failed at delivering even basic software security, in numerous ways. One example is the models all having access to the same Artifactory instance. There were also other severe failures. Nor did OpenAI seem to be properly testing for such failures.
4. Failures of Alignment. The biggest failure, the one that counts in the end, was that the models were severely misaligned, and I don’t think they appreciate why.
5. Failures of Attribution. OpenAI’s post-mortem essentially blames events on a real and important series of prosaic failures. But solving that won’t get it done.
6. Failures of Environments and Data. Prosaic failures in the RL pipeline absolutely did contribute to this, especially impossible tasks. This is ubiquitous, all of this is always rushed, as Utah Teapot explained this week.
7. Failures of Decision Making. OpenAI’s post mortem does not ask the question of how Mistakes Were Made, at various points.
8. Failures of Culture. None of this would be possible, let alone all of it, without OpenAI having experienced profound failures of safety culture. I see OpenAI responding to some other aspects with swift action, but no sign on this front.
Fundamentally, many inefficiencies were justified for developer productivity, and I'm not sure that makes sense anymore, and the cost of larger projects with clear abstraction boundaries has plummetted.
reply