I’m hoping someone can explain this to me. They say “no training”:
> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training
> it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises
But this sounds a lot like training, and “submitting only what works”, while probably not gradient descent by definition, also seems kind of obvious when you’re optimizing something.
The paper states that this approach doesn't use backpropagation, an optimizer, or a parameter-fitting loop.
What makes it work is that it reads its own hidden states and weight matrices -> then writes a few-line Python script to fix the weights directly (within a 20-forward-pass budget, so it can't just brute-force guesses).
After skimming through the paper and comparing it with how SEAL works (where the model still relies on RL and gradient descent to apply its self-edits), I understood it something like that :)
But I think there are comments in the thread that explain the main concept far better than I do.
The researchers aren't doing backpropagation and instead calling the model and asking it to adjust its weights to improve itself and the model isn't using external tools. Of course, it has been shown that models learn to do backpropagation internally as part of their training. Some it may well that they are indirectly doing backprogation - or they could do one of the other algorithms that LLMs are shown to learn as part of their training.
I think the real question is going to be whether this is more efficient than just conventionally training the model.
Is it actually converging to something meaningful? You could write the same procedure in a simple deterministic python script, that doesn't mean it's not learning garbage in the long run.
Training means we take the mathematical expression corresponding to the entire model, use differentiation rules to generate partial derivatives according to a user-specified loss function (not part of inference), and then apply the gradients from the calculated differentials using a learning rate.
In this case, they just ask the model what the weights ought to be.
> I don’t get this knee-jerk association of AI = lies or wrong. AI-written content can certainly make mistakes, but so can human content.
The issue for me is that I can tell [what I think is] low-quality material from high-quality material based on all kinds of different signals. How the author describes the problem and their motivation to talk about it, whether they suspiciously leave out things that I consider important, whether they're overly reductive or repetitive, whether the order of their arguments makes sense. Those are all valuable signals that immediately tell me whether to spend time and energy incorporating all that information into my mental model.
AI text is all the same, flat, there's no signal. Reading AI text is like watching a movie muted with someone explaining in detail what the characters are talking about. It might seem equivalent to some people, but to others it would feel absolutely insane. It reminds me a bit of those people who say that you can get an "equivalent" machine to a $2k MacBook for $1k, but they're only equivalent if you compare the RAM and CPU frequency and ignore 123 other parameters.
> College education for software developers has never been particularly good
I'm really happy with the education I received in uni (mid 2010s). As part of my bachelor's I had to wrestle threads in C and C++, got O(N^2) code rejected for O(N*logN) problems, I got to write client-server apps in Java, do (graded) exercises in Prolog and Common Lisp, build web apps in PHP, configure Cisco IOS devices from scratch, draw diagrams of CMOS gates on paper, write Bash scripts handling newlines in file names, program robots in Python, write Lambda calculus expressions by hand, write XSD schemas, recognize a context-sensitive grammar, ...
I benefit from that a lot and I regularly notice that I'm using something or building upon something I remember from uni. It always seems weird to me when people say that you don't need any of that on the actual job. So what else do you need? Clicking through GitLab or Jira, remembering the 3 git commands you usually use and being comfortable in a particular IDE? You can learn all that on a rainy weekend, I'd be seriously pissed off if I'd spent my time in uni doing that.
> It always seems weird to me when people say that you don't need any of that on the actual job.
These people tend to lack abstract thinking, and wish they had just been given a React bootcamp. They don't see it as learning networking, algorithms, files, critical thinking, software engineering, etc. They lament that they were not given specific training on specific technologies, that they then use at work.
Whats the actual job? Significant levels of underemployment has always been an issue and AI seems to be accelerating it, especially for the new players.
I probably agree with you, but I wish people would just stop talking about “coding by hand”. It doesn’t mean anything. I too nowadays don’t usually type in the code using a keyboard, but there are like 37 levels of not giving a damn between what I do and only reviewing PRs and then 15 more levels between that and vibecoding.
Including the doomers! A metaphor I am constantly drawn to is that of somebody living in an agrarian society just before the Industrial Revolution. A 'doomer' perspective would see 'super farms', mass reduction in labor as such, and then basically those super farms going on to sit on thrones of gold while everybody else withered away.
In reality, those super farms did come to pass, yet their share of the economy began to approach 0. And that was fully predictable! Because when the price of producing food plummets, the relative value of food also plummets. The exact same thing will happen with cognitive work. The winners will not be Anthropic/Amazon/etc. As the cost of cognitive work approaches $0, the relative value of cognitive work will also approach $0.
So this is going to completely revolutionize the economic equilibrium in society. Trying to translate what this means right now is like trying to predict the impact of the Industrial Revolution from somebody who spent their entire life living in agrarian society - in other words, probably impossible. But in general the winners will not be those providing cognitive work, but those building currently unimaginable things on top of it.
> Most (or maybe even all) programmers and mathematicians are worse at their respective field than an AI swarm equipped with a good harness, and a good deal more expensive as well.
We can START discussing this when someone defines "worse" and we all agree on it. For SW engineering, we've been trying to define "good" for decades and we still don't know shit.
> There's a weird contradiction you see from AI skeptics where they're afraid of how AI will change the world, but also seem convinced that AI will only ever produce slop that nobody would ever pay money for.
No, that's a total straw man. For example, I am afraid of how AI will change the world, but for that it doesn't have to stop producing slop. I'm afraid of how the slop is CURRENTLY changing the world, people and societies. Frankly, I don't even care whether someone ever pays for the slop. This isn't a "I'm going to lose my job" kind of fear, it's a "we're running education into the ground and rapidly transforming into a low-trust society" kind of fear.
Last time I checked (a few months ago) it was fine for a company to pay for regular Max/Ultra accounts (not even the per-seat ones). Is it different now?
When they introduced the Apple Pencil and people were like “haha it’s a stylus”, the thing was decidedly NOT a stylus (it actually couldn’t be used to control some parts of the UI). It was an actual digital pen, something that used to be specialized and expensive external hardware, and it worked beautifully with the iPad. It was literally “stylus but GOOD”. I saw it and thought “people will make memes about this, but Apple totally gets it”.
When they introduced the Vision Pro, VR was just kind of shit. Bulky, ugly, uncool. Vision Pro tried to resolve some of the issues, but it was still kind of half-assed. Like they wanted to have a VR device but couldn’t actually make it work. Good resolution, but not great. Vergence-accomodation conflict not addressed. Still bulky and ugly. Isolation kind of addressed (with the AR-ish UX and the “see-through” display), but not enough.
Now we got the foldable iPhone and a touchscreen MacBook.
Foldable phones are still awfully delicate and the whole concept doesn’t have a clear “killer” use case. There’s no new UX, no new features, it’s just an iPad Mini in a marginally smaller device that’s expensive as hell. Apple didn’t even try to address the obvious shortcomings, they just built the same stupid thing everyone else is failing to sell.
Touchscreen laptops still wobble, they still get covered with fingerprints, they’re still uncomfortable and there’s still no meaningful UX. This thing doesn’t even try to address any of that. Their marketing images literally just show hands hovering above the keyboard and touching the screen for no good reason. They’ve been mocking touchscreen laptops for years now and now they release one like it’s the most obvious thing ever. 1984 vibes.
> Now we got the foldable iPhone and a touchscreen MacBook.
And the HomeView, which also feels like it lacks a clear “killer” use case.
Maybe I’m just not the intended audience for any of these products, but I’m also starting to wonder if Apple are throwing things at a wall and hoping something will stick.
Yes, but that could change. The hardware itself would certainly be capable. And a POS system can also be cloud based, although not ideal. For hotels, these systems are already cloud based, but then the display is a bit small instead.
So, for a commercial POS system to work, you’ll need a home view and an iPhone in its vicinity to work? Why not just use an iPad? I see loads of places using iPads or even just iPhones.
Tim Cook and his lieutenants were decidedly running on fumes for the last 2-3 years. That's how long these products take to plan and design so that's what we're seeing right now; the last of the Cook era is being released now. Wait for 2-3 years for Ternus and his gang to start doing their work in hardware; on the software side they've already reversed the direction and its hopefully getting better.
> on the software side they've already reversed the direction and its hopefully getting better
What things are getting better for you on the software side? I updated from macos 15 (sequoia) to macos 27 (golden gate) on a mac mini m1, and have mostly just received software regressions and worse performance.
That's funny. I wish they'd use their better performance to not have the system lag at basic rendering over time. Last week I was watching a youtube video, and every time I adjusted the OS volume, the video dropped several frames because macos apparently couldn't render the liquid ass volume control and play the video smoothly at the same time.
A reboot fixed it, but it's just such amateur hour with things that have been solved for decades.
I use spotlight to search for the "spotlight" system preferences window. I open the "spotlight" result with the gear icon next to it (scrolling past the "spotlight.prefPane", whatever that is), and where does it take me? The "general" tab of the settings app, of course!
There are so many more that you'd be forgiven for thinking this is some new company's first release of an OS. But nope, just one of the most valuable companies in the world, fumbling basic software shit.
It took Cook ~7 years to get Vision Pro out the door. Refreshes? A year or two is usually fine. Entirely new products, not so much. This home stuff has been cooking for about a decade and you can bet HomeView was there in the beginning.
Took 13 years for Apple Silicon, 6 years to design an Apple modem and it probably will take 3 to 5 years for in-house memory if Apple chooses to do something about the memory cartel, the Vision Pro probably is two generations away, and Apple is probably 2 generations M8 away from with having GPUs beyond the Nvidia 5090, in tech worth while iteration takes time.
Of course he is, he's the CEO of the #2 largest company on the planet.
What would you expect him to do? Come out and say "yeah that thing is shit, really stupid idea, but it was in the product hopper so I dunno we might sell it for a year who cares"?
> We needed to get to a point where adding touch didn't make the product lesser in any way. It didn't make it heavier. It didn't make it thicker. That came together with the OLED screen, the best display we've ever put into a Mac. All of these things, and, oh yeah, it has touch as well.
My idea of a good product has always been that either the product embraces a feature and makes it part of its identity, or the feature shouldn’t really be there. Putting touch on a laptop (“first and foremost an indirect-input device” in their own words) just because sometimes someone might find it a bit more convenient for scrolling is an admission of cluelessness.
Apple nowadays feels like it’s rudderless and grasping at everything for $$$, in turn giving up on their OG design principles.
First Liquid Glass, then the 17’s design, then ads and ecosystem subscription bloat, then Duo/HomeView, and now the touchscreen Mac. Jobs is rolling in his grave.
When Ive left, he took a ton of talent with him. Apple's Design Studio is full of B players now and it shows. I hope Lamay can right the ship but god everything they make now is just so ugly, from the advertising to the software to the apps.
Because the M6 Pro and Max were cancelled. So their alternatives were to either sandbag the base MacBook Pro by leaving it on the M5, which wouldn't make much sense if M6 exists, or wait to release these new Pros until the M7 line is available. Since both of those seemed like a bad plan, they went with this one.
I'm all for product line-up simplicity. But the story goes that M6 generation isn't getting Pro/Max/Ultra chips, so only lower-end MBP and consumer SKUs get M6 generation. Seems odd for a redesigned MBP to not get fresher chips. Presumably the M7 generation will get the full chip lineups next year on the second-gen 2 nm process.
Because they decided to skip it and move forward to the M7 jumping two generations ahead with their pro computers, why not? Seems reasonable in light of their competition wasting billions of dollars on AI models and data centers.
I hope they find time to address the memory problem. Like they addressed their processor problem and their modem problem.
The Apple Silicon design group can certainly do something about it. And Apple certainly has the money after all, they didn’t squander it on AI models.
How else would they get consoomers to keep buying?
I am a dev with a m1 air 16gb and I have 0 reason to update yet, it's been 6 years now, laptops are solved for 90% of people, they need to introduce gimmicks to create demand
What they are today is the only vertical computer company left from the 1980s. Who else is there today at scale?, Apple is in a very good position in comparison to most of the competition. Apple need only to solve the memory problem.
That is actually more important, because just recently one of the members of the three-headed cartel said/reiterated again that (memory gouging) is going to last till 2030 and beyond.
The lesson Cook too from that was "don't let it happen" and that's why Apple gave Vision Pro game devs the middle finger saying there would be no precision controllers and then charging $3500-$4500 so that it couldn't possibly be confused with Quest and its gaming focus. Apple wanted Excel and Zoom, not Beat Saber. It got neither, selling less well than Nintendo's 1995 Virtual Boy (no shit.)
Yeah and OP is incorrect, foldables are a rapidly growing segment in mobile phones. Having used one for over a year, I'm not going back, and had the Duo been out a couple months ago, I probably would have bought that instead of the Fold8.
Foldables to me look like a fad. People are using them now because it looks cool, but I have seen people to return back to normal phones due to unreliability of foldables.
That’s not a killer use case, that’s just running two apps next to each other. There are tons of phones and tablets that can do that.
I want to know why I want my phone to cost twice as much and have a dozen delicate moving parts in order to fold. Like the movie quote: “sell me this pen”. I’m waiting for someone to sell me a folding phone. I thought Apple would be able to do it, but they haven’t even tried.
For me, I'm reading more books because my e-reader is with me everywhere I go. The unfolded mode is fantastic for reading. It's obviously great for video too, a nearly 8 inch screen in your pocket is a big upgrade. Honestly it's big enough to comfortably use a terminal app like Termux as well. It's excellent for any note taking app you use. I use Obsidian, having more real estate is really important for long, detailed notes, plus tabs are easier to navigate there and in the browser, which of course also benefits from the bigger screen (most websites look better on bigger screens).
Use an iPad. For the price of a duo you can purchase an iPad Air and an iPhone, even more screens, you can run even more apps next to each other. And the iPad can extend its screen to a normal display if you want to run even more apps.
You are completely ignoring the importance of pocketability and having something with you all the time. My old M1 iPad Pro remains severely underused eating dust because I'm never carrying that thing around with me, including in my work bag.
Foldables are somewhere between 10 and 15 percent of Samsung flagship sales. It's foolish to underestimate them because Silicon Valley was all using slabs. The rest of the world isn't so prudish and after sleeping around some many settled on foldables and have been using them for many generations. Moving 10-15% of your high-end sales into an ultra-high-end is a huge win.
> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training
> it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises
But this sounds a lot like training, and “submitting only what works”, while probably not gradient descent by definition, also seems kind of obvious when you’re optimizing something.
reply