I think someone who likes AI thinks it’s good, someone who doesn’t, doesn’t think it’s anything impressive, and concretely nothing of any significance has happened to the world, certainly nothing like the crowdstrike shutdown so I don’t find the capabilities to be anything impressive.
> and concretely nothing of any significance has happened to the world
I would hypothesize that all the long-held "easily discovered if you look in the right place, but never reported and so never fixed" 0days in operating systems that criminal gangs and militaries have been relying on (to do cyberattacks with, unlock devices for forensic data recovery with, etc.), are increasingly being exposed and cleaned up, much faster than new such exploits can be discovered; and that this may have given an edge to whichever side of these conflicts don't rely so heavily on hacking the other to achieve their goals.
It’s true that I’m an AI adopter. My psychosis is pretty advanced. I’m running four clankers in parallel right now.
The 9 bugs I got were clear security bugs under the Fil-C definition of memory safety. One reason why my experience could be an outlier is the effort I’ve gone to defining what the security properties of Fil-C are, to the point that there’s a crystal clear falsifiable boundary. So the bugs aren’t a matter of opinion or judgement. That helps clankers and it helps me when screening the bugs.
> concretely nothing of any significance has happened
This is a good point, but “cyber apocalypse dooming is an exaggeration” is a different statement than “bugs found by Anthropic aren’t real”. I agree that the dooming is an exaggeration. I don’t agree that Anthropic’s bugs suck.
Two reasons why bugs-found-by AI-are-really-scary and nothing-is-happening can be true at the same time:
1. Anthropic is sharing the bugs found by their most capable model with defenders, not attackers. Defenders are working overtime fixing those bugs before the embargo runs out.
2. The attackers that have access to Anthropic-class capability are using their bugs surgically. They aren’t using them so widely that folks would notice. That strategy is business as usual for advanced attackers
I obviously don't know if this applies for the Fil-C bugs but what we saw in Rust is that there's plenty of "technically that's not OK" which is very distant from "this will actually cause real world problems" https://forge-blog.github.io/posts/rust-std-memory-safety/
For example it's definitely not OK that you can impl Pattern in a way that causes non-UTF8 data in a string with safe code. But you probably didn't let random attackers impl Pattern, on a nightly Rust where that's even possible, and so that's not going to cause problems in our real world - even though it's wrong and must be fixed.
I'm speculating out on a limb here, but maybe the Mythos zero-days that were developed into working exploits involved a few of those "usually won't cause real-world problems" happenstances.
A model with all the world's knowledge might know exactly how to weave them together into a successful exploit. And if tomorrow's models are even more capable, perhaps what we're experiencing now is just a fire drill for what's coming.
> One reason why my experience could be an outlier is the effort I’ve gone to defining what the security properties of Fil-C are, to the point that there’s a crystal clear falsifiable boundary.
Building software boils down to defining things correctly. So, it sounds like you did most of the work here. Imagine that you took your definition and moved it into a formal specification (programming language) and then ran that to verify the security properties of your system. What's different?
Or, imagine that you took your original code as your specification language and then iterated on your understanding of it until it was correct. What's different?
Attention highlights and resolves bugs, so that's the baseline. In order for AI to get value, it has to be some kind of multiplier on your attention, but the reality is generally the opposite.
I wonder what it's like maintaining a project where you regularly get competent contributions. No wonder they are less impressed with AI - they are the 0.1% of maintainers who had something better.
I see it the other way; Linux is a huge project, but being a solo maintainer for a mid-sized project like Fil-C is a different level than one of dozens of maintainers for a large project like Linux. Sort of like being a solo founder for a semi-successful startup is different than being a rank engineer at Google.
I did an analysis on the code I'm currently working on pre-AI and post-AI. There is no appreciable difference in implementation code size before and after, but the total LoC has grown substantially from having a lot more test code.
So you're right. Without AI the average lazy developer would write 50k implementation lines of code and 0 lines of test code, and now they have 450k lines of test code in addition to their 50k of implementation code. It might not be great test code, but I know which I'd still rather have (hint: the tests).
I strongly disagree AI generates codebases of the same size as humans excluding test code. You can just look at Codex, codebase that is almost millions of lines of Rust Code (source) for a desktop app/ Terminal UI’s. I saw a lab employee somewhere cope on X by saying their terminal UI is as complex as a game engine, but last I checked, they just need to generate some 2D pixels, not entire 3D worlds with physics and sounds that is done with ~500k lines of code.
LLMs are laughably bad at writing code in some languages and shockingly good in others, so I can see that being the case if you chose poorly. But, especially when you have decided that you aren't even going to look at it, why would you use a shitty language?
The average person uses social media, and Microsoft office suite. They really don’t require much else. Social media doesn’t get disrupted by software, and MS Office was already disrupted by Google suite, if you didn’t make the transition then, you ain’t gonna make it now.
20 years ago the average person used a hoe and a cooking pot on three rocks over a wood fire. Now they use WhatsApp, and in a few months they'll be using ChatGPT, but not Microsoft Office. But I wasn't talking to the average person. I was talking to djeastm.
Not sure what your cohort for "average person" is, but 20 years ago was 2006 not ancient rome. We had avoided the millenium crisis, and I was playing Counter Strike against people over the internet.
My reference group for "average person" is people. (What else could it possibly be?) There are 8.3 billion of them now, and there were 6.7 billion 20 years ago. 02007 (19 years ago) was the year when the median person transitioned from living in the country, either on a farm or herding livestock, to living in a town or city: https://en.wikipedia.org/wiki/Urbanization#cite_ref-urbaniza...
One side of my family has lived in cities and towns for over 100 years, but they were not average.
Could you even call it "working" for video games, based on the result, for the same kind of reasons you'd say it doesn't work for those other media?
(To qualify the rest of this, clearly it's job done in terms of the author wanting to make something memorable for their nephew - it's quality uncling, with the best of intents - but in the context of how what's been shared is going to contribute to the discourse around AI, in the shadow of a new model release, during an ongoing pre-IPO hype bubble...)
I'm seeing something which has has the surface-level features of a belt scrolling beat 'em up, but it's missing the result of all the different types of work that would normally happen through "make it actually interesting and enjoyable to play" design and iteration, e.g. you can walk through every stage unchallenged mashing punch, there's no evidence from the 2 stages I played of an interesting mix of enemy types with unique AI which can combine in ways which add challenge and variety to a belt scroller, you can stand and basic punch the first boss to death stepping aside only for its AOE attacks, the second boss infinitely spams a running move when it reaches low health and sometimes bumps into the edge of the screen behind it.
_Something_ was one-shotted, sure, but what we expect of a belt scrolling beat 'em up that someone cared enough to finish and release wasn't, not that people who will point at this in the near future as evidence of "OMG new Fable can one-shot games" will care.
Honestly aesthetically, good human code is always >> Opus 5.5.
For example, recently Opus 5.5, in some multithreaded app I was working on at work, used complex timing analysis with hardcoded time values 1.2, 0.95 etc to ensure that some activities only happen in a sequence (A->B->C). I had to tell it to just serialize these activities by putting them in a buffer. Both versions passed all tests (I’m sure it wasted a lot of time tuning the first set of numbers to pass the tests, but no human would do that nonsense). Increasingly, you feel it’s thinking diverging from humans more and more as the labs lean into RL.
> We need the best and brightest if very country to stay in US.
These are not coherent positions at all.
Frankly I think we need the best looking people, not the most talented/ hard working/ intelligent people anymore, because that might be the only thing AI won’t replace for a while (decades).
I think for small snippets and well defined codebases, we probably can’t,
But if we say a fully vibecoded project vs handmade, I suspect there are AI tells outside of basic stuff like bar names etc that a model can learn, and it will be fun to know.
We could probably also do these for maths.
I guess what I’m getting at, is even in these days extremely verifiable fields, I guess AI has diverged enough that I think we can train a detector to separate AI from how a human thinks?
The problem is the technology. I get very well paid in my tech job and can live forever on my FAANG salary, it’s not about the money:
There are 2 types of tasks people do throughout their life.
1. Creative tasks that provide self fulfillment
2. Uncreative monotonous menial tasks that degrade life.
AI takes 1 and ruins it to 2. The jury is out if it will fully take your job, I don’t know if that’ll happen but no one asked for automation of these creative tasks.
When you point these AI bros talk nonsense like cure cancer, Dyson sphere, China bla bla bla.
At the very least, models need to be post trained to stop outputting art, this will never happen though, meanwhile I will enjoy my half a mil a year, while all these AI bros lose their mind over graphs, loops or whatever nonsense they’re concocting up.
Nothing about this is inevitable, AI is as inevitable as nuclear, it seems to require even more capex than nuclear.
I think someone who likes AI thinks it’s good, someone who doesn’t, doesn’t think it’s anything impressive, and concretely nothing of any significance has happened to the world, certainly nothing like the crowdstrike shutdown so I don’t find the capabilities to be anything impressive.
reply