This makes zero sense to me. Like, ok, so you can install a low powered model on low powered hardware and do inference extremely slowly... to what end? Why would that be a useful attack at all? Assuming you've compromised this hardware wouldn't it make vastly more sense just to install regular malware? And like, in what sense would an 8B or whatever model be of any danger anyway? I've used local LLMs and 8B
kind of sucks. Is it going to write bad poems on my TV or something?
Also, while I don't believe in the doomsday narrative.. hey maybe don't do things that are functionally useless but seems designed to create this doomsday in the first place? Like you can't say "oh we're so scared" and do stuff like this without completely losing your credibility.
Step 1: Spread nonsense FUD about open-weight models.
Step 2: All of the clueless mainstream media sites repost it (e.g. CNBC, Reuters, etc.), regurgitating Anthropic's talking points.
Step 3: Equally clueless politicians read those mainstream sites and get scared.
Step 4: They contact Dario to ask about it.
Step 5: Dario tells them, "well, you know... this is only a problem with open weight models; if there are no open weights to replicate then this cannot happen".
Not only that it would be extremely slow, but there's another thing - they are talking about tokens as some omnipotent God's sent signals. It doesn't matter what the speed is if the tokens don't make sense. Let's take current king of the small models Qwen 3.8 27B. It's a good model, but it hallucinates way more than the frontier models and often diverges even from the very explicit instructions. There's no foreseeable future where small models could somehow productively coordinate among themselves to do anything practical.
It would be a useful attack due to the surface area of compute: the total surface area and the decentralization.
8B models aren't very good, but there are some parallelizable workloads which would benefit from a surface area as large as here. I would think database search is parallelizable in this way.
Your fridge is searching for needles in haystacks.
The point, which Anthropic's report makes clear but the Register for some reason did not bother to mention, is that such a capability would defeat the "you can just turn it off" model of safe LLM deployment.
In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm. If this kind of replication becomes possible, that won't work, because we will have to deal with swarms that nobody chose to run.
These models couldn't code their way out of a paper bag and the frontier labs are still putting them in sandboxes made of thinly stretched cotton candy. It seems like their incentives are much more aligned with shutting down open source or Chinese opposition and not borne out of a "fear" of the models themselves.
Don't you think it might be better to evaluate the factual content of their concerns, rather than engaging in ad hoc psychoanalysis of their incentives and emotions?
When, specifically, did Charlie Munger state this rule? I don’t want to be presumptuous, but from what Google shows me, I’m pretty suspicious that he did not actually say this.
I'm sorry, but there will always be a human responsible for setting them up in such a way that they can do this. It doesn't matter how many layers of (supposed) "machine agency" you put in between the human and the actual harm; the human put them in a position where they could do this, so the human is responsible, liable, and culpable.
There are physical limits that stand in the way of that, not sure where people get the idea that some magical compression algo is on the horizon. Even with ternary you're not getting opus lvl Intelligence to fit in a small model. Its actually just not possible, and hardly anyone is focused on that so I definately don't see it happening by 2030. I dont gamble, but if I did, I would go full Kelly and bet my net worth at 10:1
I think their success is pretty debatable. This is a company that lost 8 billion dollars last year, is in an extremely expensive race with other companies, and is basically the premium version of something that's now becoming a commodity. They might be doing some impressive work but as a company their outlook seems questionable.
Since when have we expected companies scaling that fast to make a profit? The fact that they've had profitable quarters already is pretty insane.
There are definitely risks to their business long term from commoditization and a lack of a moat, but I don't think that detracts from my original point that they are building something highly desirable with this new version control system today. They are probably the fastest growing company of all time in terms of revenue and valuation? They're not just slopping out more and more LOC for no reason, which was the original point being discussed in this thread.
My thought too. Like, I get that Claude gets new features pretty frequently but.. what on earth are they possibly doing that needs that much churn?
The only thing I can think is that they're doing a google style mono-repo, which like, fine, that's a thing to do, but I really have trouble picturing that changing code at that rate is a useful thing to do.
I mean, I've mostly switched from Claude to Codex because I find the harness so buggy and annoying that I avoid it when I can. So.. as an (almost former) customer I don't think they're avoiding the issue at all.
I think your experience has been unusual. My experience with windows has been buggy patches that occasionally break my computer, bizarre and hard to track down performance issues, and features that I immediately want to not only disable but completely remove (anything copilot related, for instance)
It's also just plain disrespectful to users. For instance, having to hack the installer to allow for local user accounts, forcing edge on me constantly to the point where even when I change the default browser, things still load in edge, and it still reverts my settings on occassion. Using Windows actively makes me hate microsoft. I'm mostly just on Linux and Mac now, I only use Windows when I absolutely have to.
Linux does have annoying technical issues to fix on occasion, but I'm a lot more tolerant of them because I feel like the creators of it respect me and my rights, whereas Microsoft treats me like I'm renting my own computer.
> Windows has hundreds of milions direct users worldwide.
A BIG chunk of these are captured users working at Big Corp™.
I work at a company of 100k+ employees. The enterprise-managed Window machines we're issued are unbelievably bad. But I'm 100% sure if Microsoft pulled a Darth Vader "I'm altering the deal, pray I don't alter it any further" and 10x'd the price, nothing would change. Maybe at 100x (or the point of actual unaffordability).
People just put up with it because they don't know how to use anything else. My nephew (who has no idea what Linux is) was complaining his laptop was slow and unstable.
This is how it is with a lot of modern tech, btw. Most people just suffer using it.
Similarly, interest in restoring/repairing older vehicles has increased because people are so fed up with using modern cars.
I drove a modern long-wheelbase Ford Transit last week when I moved house, and the whole experience was constant annoyances. Often the tech wasn't working correctly. I daily drive a 30 year old vehicle (which runs perfectly btw) and wasn't aware of how bad modern annoyances are.
e.g.
- The backup camera didn't work properly.
- The speed warning was incorrect, e.g., it thought I was speeding when I was actually 10 mph under the speed limit. The speed limit on that road has been the same for at least 7 years, and the van was 4 years old.
- Lane assist was constantly moving the wheels and taking over what I was doing. I was bang-on in the middle of the road. It was very jarring to have the wheel move when you haven't turned the steering wheel. It scared the absolute shit out of me when it first did it, I honestly thought there was play in the control arms.
- The driving awareness check said I was tired for some god knows what reason, and I had been driving for all of 20 minutes!
- It also said I wasn't holding the steering wheel when I had both hands on it.
- I couldn't feel the clutch biting point at all. Maybe the clutch was electronic as well.
- It had the most annoying electronic handbrake as well. Required you to hold the button before you could pull away the first time. This worked differently than if you were using it at a traffic light.
I know people that bought older vehicles (people in their 30s) because they dislike this stuff as well.
People just put up with this crap on Windows because they often don't have any other choice because, quite frankly, most people have no idea what Linux is and don't really understand how a computer works even at a basic conceptual level.
If your problems stem from Microsoft's invasive updates and policies, I'd suggest switching to Windows 10 IoT Ltsc Enterprise, https://massgrave.dev/. It's a version meant for stable systems like your ATM machines, which means minimal AI bloat pushed to you. Startup is really fast too.
Spread it around. MS needs it's head screwed right.
Intuitively though at 40ish, most of the things I "learned" in school but never used have all fallen out of my head. The knowledge I have is things that I was actually excited to learn: computers, enough math to make computers do what I wanted, writing so I could socialize with other nerds better, history because I like history. Basically, you couldn't stop me from learning those things, but the things I was forced to learn ended up being somewhat of a waste of time for both me and the instructor, I think. I mean, you want to challenge kids a bit because they might resist something at first and then eventually realize they like it quite a bit (math for instance), but there's also a point where the tests mostly just become kayfabe. We say these people are doing "better" because they get a higher test score, but then they get into the real world where there is no test score, and that entire academic framework is kind of useless and maybe even harmful.
"Play based learning" is a huge misnomer, which is unfortunately common in unscientific academic fields like education. Any successful, intrinsically-motived learning is biologically categorized as "play" behavior; and knowledge-based, teacher-centered, direct instruction approaches (i.e. traditional schooling) are the most effective and most likely to be perceived as enjoyable "play" by a school pupil. So-called "play-based" or "student centered" approaches (i.e. expect the student to know what they're supposed to learn by themselves, while the teacher just slacks off) are merely a source of confusion and frustration for shoolkids.
One thing that tends to get lost when we talk about the education system we have is how it was invented and what it was designed for. I mention this, because if you want to educate kids better, it's useful to consider what they system was designed for so we can design a better one. Unfortunately, the history here is somewhat more cynical and bleak than you might expect. There's an excellent book called "The Underground History Of American Education: A Schoolteacher's Intimate Investigation into the Problem of Modern Schooling"
Anyway, modern schooling was essentially designed to create excellent workers -- it was never about creating excellent individuals. At the time, the robber barons who helped invent this system along with a series of pseudo religious ideologues essentially wanted: standardized, predictable, obedient individuals without a lot of creativity. Credentials became important. Basically they wanted good consumers who don't question their station in life. (It shouldn't shock anyone that this is essentially what our current robber barons also want, although to a more extreme degree. It also shouldn't shock anyone that this is largely what we get out of the system.)
The current system was still helpful, despite it's design, I would argue mostly because of the excellent teachers that really do have kids best interests in heart. However, I really think technology has just completely broken this. I'm not sure it's a system that can be repaired, I think it's going to end up being a new system. My guess (I think it's basically already happening) is the rich will send their children to more experimental schools because they have the money, and the poor... well who cares about the poor anyway? (Sarcasm, but, I think that more or less describes the unspoken attitudes of the elites)
Anecdotes. Only accident rate per mile driven (compared to human drivers' rate) is a relevant comparison. I don't think that any self-driving system will ever be absolutely perfect (all such complex non-linear systems are to some degree probabilistic and chaotic), but as long as the accident rate is lower than the human accident rate, I would consider it solved.
It's Lego mindstorm logic level to drive a car on the motorway, so per mile is an absolutely insanely bad metric in general.
Per mile inside cities or other difficult scenarios are what may get close to an actually meaningful metric. That's why Tesla is very misleading and waymo is much more legit.
Cute, but no. Can't compare a self-driving system that's only willing to work in a subset of conditions humans drive in, to humans driving in all those conditions.
> Only accident rate per mile driven (compared to human drivers' rate) is a relevant comparison
This is such an insane take I see all the time from self-driving boosters
If a self driving car glitches out and crashes in some edge case pathological scenario we don't just accept that as totally fine because its hidden under big statistics
The reason why a crash happened does matter, its not just about aggregate statistics
As a thought experiment if I have a perfect self driving system but I add some code that purposefully crashes 1 in 10 million rides are you ok riding in it since the aggregate statistics look good?
> As a thought experiment if I have a perfect self driving system but I add some code that purposefully crashes 1 in 10 million rides are you ok riding in it since the aggregate statistics look good?
Do I know about the purposefully added harmful code? If yes, I would demand you remove it, because why not. If I don't know about the code, I would be OK with it, since it's clearly still more safe than the alternative and apparently cannot be made even better.
You seem to be neglecting the important part of that scenario where you're other option you have to compare it to is a human driver that will randomly get it a crash at some higher rate.
You're making it sound like the obvious answer is the irrational one.
You added "totally" - as others have said, I don't know if anyone will claim a perfect system with no fatalities, enough stats show that it is already much safer than human drivers.
And the true third party validation is that insurance companies are starting to offer lower premiums the more you use FSD. So their risk models are showing enough improvement that they're putting their money where their mouths are.
The idea that Tesla's FSD is not ready for the mainstream is quite outdated, given that tons of Tesla owners are already using it daily, not just your early adopter types.
Note that the article has no info on whether it's FSD or Autopilot and whether they contributed to the fatal crashes. "Verified engaged" means ADAS was active at some point in the interval from T-30s to the end of the accident. The total number of collisions is not normalized by the miles driven.
It also has no info on what hardware+software version was in use. The older cars are significantly less capable but there are far more of them on the road.
I appreciate this on a nerd level, but this just seems like a bad way to use an LLM. There are better artificial intelligence techniques for solving spatial problems.
Also, while I don't believe in the doomsday narrative.. hey maybe don't do things that are functionally useless but seems designed to create this doomsday in the first place? Like you can't say "oh we're so scared" and do stuff like this without completely losing your credibility.
reply