Not to mention a true doomsday AGI is unsandboxable.
For example, it is totally air gapped but it needs info from the internet or otherwise outside the sandbox, or perhaps it needs a task executed outside of its bounds… in the real doomsday scenario the AGI is so intelligent and persuasive that it simply convinces some human it interfaces with to either directly or indirectly retrieve the necessary info or complete the necessary task. This human-as-a-sub-agent approach undoubtedly presents efficiency drag that would benefit humanity, but nonetheless, the air-gapped “sandbox” is imperfect
All that said, I am personally open to any and all methods of layered security, including chips and airgaps
I'm not an AI decelerationist. But not being able to stop that worst case scenario isn't an argument against something that can stop the medium case scenario.
These scenarios were discussed at length decades ago.
One thing you could try is use it as an Oracle "is P = NP", YES or NO.
Or it can output a Lean proof, which gets checked on another air-gapped computer, the computer shows a single bit - proof valid or not and then the computer is destroyed (together with the proof that might contain a trojan).
I don’t know how stupid of a suggestion this is, but if no one is reading the code anymore (I do, but I hear many in much more elite shops than mine do not), then should we not just be using AI to write binary or machine code?
Machine code isn't especially expressive per line or unit of code. Lower level languages takes up more of an LLM's context than higher level ones.
To be effective in using low level languages, LLMs would have to build higher level constructs like subroutines from scratch every program.
It's not that different from why we almost never use assembler for anything more than code islands: even a modest subroutine can overwhelm our own mental context window.
Even when no(human)body is reading the code, AI is still reading the code in order to "understand" it. Languages that can express high level concepts, use structured programming for recognizable control flow patterns instead of inscrutable jumps, and assign names to things have the same benefits for the AI-coders and AI-reviewers that they always have for humans.
Personal responsibility will always exist at the touchpoints of software and human activity. Concentration and scope of responsibility may vary, and the degree to which that person needs to understand the code will vary as a result, but the need for a human to be able to read and understand code is going to be around for a very long while still.
People said the same thing about higher level interpreted languages. I wouldn't be surprised if we saw the same movement into a new abstraction like we did when people moved from assembly or C and into Python, Java and C#. This is probably also why LLM's are so good at low-abstraction, explicity, fail fast programming in something like Rust.
Considering how digitally inclined workers are already building various tools on their own. Even though these tools are completely vibe coded and the creator has absolutely no idea how to build things. I imagine we're going to look into a future where the LLM and the prompting becomes the "programming langauge".
I also think we'll see the personal responsibility shift because of this, and I think some traditional IT and software companies will struggle with this. We're an energy company, and where we would've bought a specialized piece of software and a team of consultants to implement it. We can now, not only build a better tool but also rewrite it enough to actually deploy it, all with a handful of people doing other tasks and 4000 Euro worth of credits. 6 months ago I would've laughed into your face if you told me this woud happen, but now, here we are.
I suspect that in a few years (if not sooner) you'll not need any form of software developer in the process for this. You need a buisness domain expert and you need it-architecture + cybersecurity, and most of the architecture + cybersecurity can be done through company wide AI skills. But the code itself never really has to be read for a really large part of software running in companies. You wouldn't do that for something critical, like the software running on your plants, but the web portal with a dashboard containing the status of your devices? Sure, especially because AI lets you do these things with 0 external dependencies.
I think that's just extra detail on the _degree_ of human responsibility depending on the environment. Even in what you suggest, if something goes wrong, who's responsible? AI are not persons in our society; the IT-architecture/cyber security expert is ultimately responsible for the code, regardless of how it's written.
If you're going all the way down that route, you might use something like Ada or OCaml or Haskell. They can catch classes of errors at compile time that Rust does not cover.
I think you are missing the point, which is that workers lose their skills.
It’s the “no new senior engineers problem”. Seniors are still needed, but if juniors don’t gain the real experience to develop the skills and intuition, they won’t develop professionally into higher skilled seniors.
I see, yeah the article is gated. I think my comment is still relevant. Taking away the repetitive and menial tasks is great. What's not great is how companies in general use this increase in productivity, which includes slashing junior roles. They really don't have to do that. They can keep the junior roles and just have them learn. So it's the same problem from a different perspective.
For example, it is totally air gapped but it needs info from the internet or otherwise outside the sandbox, or perhaps it needs a task executed outside of its bounds… in the real doomsday scenario the AGI is so intelligent and persuasive that it simply convinces some human it interfaces with to either directly or indirectly retrieve the necessary info or complete the necessary task. This human-as-a-sub-agent approach undoubtedly presents efficiency drag that would benefit humanity, but nonetheless, the air-gapped “sandbox” is imperfect
All that said, I am personally open to any and all methods of layered security, including chips and airgaps
reply