Isn't this what openai is doing, at least in spirit, by running multiple passes through the network before exhibiting thinking tokens? Isn't the same issue one of safety and control, because being able to interpret thinking tokens helps u control / monitor unwanted behaviors, and multiple passes before writing out the thinking tokens increases their capacity for deception?
One of the big points is that looped architectures don't necessarily remove CoT style reasoning. They definitely change it, but the model can still be prompted for this kind of output. It just spends more computation on individual tokens. I think the hope, in that particular scenario, is that the model spends less tokens on things like backtracking.
But there are lots of approaches to looped transformer architectures. Some have some kind dynamic loop (i.e. they might loop until they satisfy some kind of equilibrium, or they might have some kind of routing function that makes different tokens loop differently, or any number of other approaches). Others are simpler "loop this many times" architectures. It's a really active area of research with some pretty deep roots, if you want to dig in! This paper blew up a while ago, and it's a looped transformer architecture (though not like the ones discussed here) that isn't autoregressive, as an example: https://arxiv.org/html/2510.04871v1
I could imagine that you could loop over and over, but direct the outputs of each layer to two destinations: a) to continue through the layers, looping, and b) a separate CoT text output layer to make those CoT concrete. In fact, the model itself would not need to be aware or have access to those CoT traces at all. I am not an AI guy, so I don't know for sure, but that seems reasonably possible.
Looped transformers are entirely equivalent to running a single pass through a much deeper network with shared/repeated parameters. There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one. An objection to looped transformers "reasoning in latent space" is ultimately an objection that applies to any kind of efficient scaling; it's also worth noting that there's been successful work on extracting ex-post natural language descriptions of these internal latent states, so it's not entirely correct to describe them as inherently non-surveyable.
> There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one.
I certainly think you could say both of those are reasoning in latent space. But of course people can have different definitions of what they consider reasoning. Maybe this is because I think it's hard to pin down what "reasoning" is so I instead alias it to "computation" which both of those examples do.
There are a huge number of citations about looping on this Ouro page [1]. Ouro is an open looped LLM with a very instructive paper at [2] (also containing references to lots of helpful papers). Basically they get these four results from looping:
• Parameter efficiency. 1.4B and 2.6B parameter looping Ouro models match 4B and 8B transformers on most benchmarks.
• Adaptive computation. Looping is trained in a novel way to avoid exiting too early or too late.
• Reasons to use looping. Looping does not increase raw knowledge storage (approximately 2 bits per parameter for looped and non-looped models) but dramatically enhances knowledge manipulation capabilities.
• Improved safety and adherence. They find that safety and alignment of CoT to final output improves as recurrent steps increase.
Hmm, but is it not the case a much larger model could potentially reason more in latent space, depending on how it's set up?
I thought a looped transformer is in fact reasoning more in latent space, though "reasoning" in a somewhat different way from the standard reasoning token approach.
I don't think this is true. The way I understand is that there's
1. coarse grained processing
2. fine grained processing
Tokens are coarse grained processing. While I agree that during a token generation, that processing is fine grained.
But Looping is fine grained processing or reasoning and it is not limited to discrete tokens within passes. This is qualitatively different because a lot more "stuff" happens between tokens. You are right that it is similar to a larger model but then even COT is similar to using a larger model.
I'd be happy to be corrected but my understanding is that it's just a side effect of scale. If we do more computation before we output, it becomes less observable.
There may actually be a somewhat of a way to establish "what" is happening during these latent-2-latent communications by observing the latent geometric vector "attractor basins", and maybe pairing vector clamping(?) to some of these (unnamed)SAE Dictionary libraries.
There are some safety/alignment attractor basins that are showing some signs of potential importance(such as scale-invariance) with the current/future prospective systems moving forward, and an ever increasing peak utility regarding such architecture(s).
An addition of some kind of, "spoof injection reclamation effort" might also supplement this, as well as, some ways to shunt the "dragging" effects from adversarial timing/fatigue attack(s) down at these interaction scale(s)/level(s).
Very similar. Might even say it's the same thing viewed at a different angle.
We don't know the details of OpenAI's implementation, mind. But the "limited layerwise depth" problem was known for a while now, and both OpenAI's approach and this are getting at it. And that's far from being the first attempts to address it.
Training and stability were the main issues that stopped it from being adopted rapidly and widely the way something like MoE was. But I think it was always a matter of time.
- but you'll never be sure the two systems aren't colluding to hide things
- yet, even for layers that don't output text directly, you can find meaning in weights - there's an old example where you provide the LLM samples like 123, 321 and sampling from the middle layers lets you find the word "reverse" which is not in the output
Anthropic has published about how to do this - basic plan: train an interpreter that can go back and forth from English to neuralese on the round trip loss. Pretty smart - they’ve published for a bunch of open weights models.
I've only played a handful of these games .. but do they work together as a theme or story progression or are they, as I suspect, pretty much independent rehashes with no real relationship other than making money and appeasing fans?
> no real relationship other than making money and appeasing fans
That's a very cynical and dare I say Hackernews take. You could also, less cynically, see them as unique pieces of art that have managed to carve a space in a field where, by necessity, a lot of money is needed to create it (AAA games).
Tears of the Kingdom is art. It's a beautiful, simple fairytale told through the medium of a huge open world game and there's something unique in being able to embody that story for so many hours. I haven't played any other Zelda games from the past ~20 years but if they're similarly artful (and by all accounts they are) we don't need to be cynical towards their simplicity or the fact that they're variations on a theme. Pretty much all human art (and business) is, after all.
Which is the correct way to design a game, I am not going to go full Carmack and say the story does not matter at all, but at their heart a computer game is a simulation, a complex machine full of intricate moving parts. Getting thees parts to mesh well is the most critical part. and the story however well or poorly written sort of sits unconnected on top of it, for the most part ornithological to the machine underneath. The bit you actually interact with.
It is why I consider excessive use of cutscenes a tell of poor game design, They stop the machine and pull the player out of the environment. They have their place, but generally should be avoided.
book: tell a story
movie: show, don't tell (big sin opening lore dump)
I like "do, don't show" but I don't think that precludes a thin layer of gameplay over a story. Walking simulators are a prime example. The entire point of a walking simulator is the story, but the game doesn't progress without input from the player, unlike in a cutscene/movie. What Remains of Edith Finch is one of my favorite games, but the gameplay is pretty barebones.
To the exception of TOTK, I _never_ noticed any link (hehe) between different titles. I know there are some references, but I never catch them.
Even Majora's Mask, which is an official follow up occurring months after Ocarina of Time have a totally different story to tell that don't even try to continue the OoT story and is more just another completely independent story barely in the same universe.
I love the Zelda games, I consider myself a fan. But they are just that : good games. Each of those games do have interesting stories to tell, interesting universes to visit, but the overall "franchise" lore is totally uninteresting.
And honestly I don't think it's something bad, it means each game is free to go where it wants, and every game is free to have an universe as rich as it wants.
I see Zelda totally like I see mario : it's just universe codes, not a real lore. Nobody cares if Mario Galaxy happens before or after Mario 64 or Super Mario World. Zelda is the same for me. And that's cool.
Even TOTK which is meant to be a direct sequel to BotW is...only loosely connected. Like obviously the map is the same and characters kind of reference some of the events from BotW, but (frustratingly) it's like almost everyone underwent extensive lobotoby between the games and forgot 99% of what happened. The entire lore of the world gets rewritten too.
From what I understand - it's because Nintendo famously designs the game and gameplay systems first and then writes a story to match what they built. If the story doesn't work with the setting then... - it's still a good game at the end of the day.
They do have common themes. Sometimes, they (seem to) explicitly refer to each other, and sometimes, they make vague allusions to things you might interpret as having happened in other Zelda games. But overall, I think it's fair to call most Zelda games independent of each other.
I did not know that about laser fault injection, at all. What practical uses has this technique found? Analysis of foreign military hardware? Cracking into secured systems?
You know how android has safetynet? And how they're planning on making it so you can't run your banking apps unless they've approved your operating system?
Safetynet is a complex dance of crypto-keys, involving your phone and googles server. It generates a key from whatever state your uninitialized ram happens to land in (which is pretty consistent per chip and is a good source of randomness), uses that to generate a key, uses that key as a chain of trust to tell google whether or not you've modified your phone. It's some complicated stuff.
But if you can extract that key, or the source of data used to generate that key, you can trick safetynet fully and run your own software again. So how do we get that key?
Well maybe we can stop the safety processor from clearing that memory so we can read it? Glitch the chip when it tries to clear memory, so clearing memory fails. Or maybe we can glitch the part of the chip that keeps you from reading the region of memory where this finished key is kept, after it's already read that memory.
Basically you can use this kind of attack to make a chip sometimes fail when it goes to do or check something, including checking whether you should be able to extract a key.
This technique applies to reverse engineering as a whole, whether it's a state level actor reverse engineering military hardware, or you trying to run your own software on a device you own.
---
Note the android example isn't practical, you'd need to like decap the main chip and a whole lot else. This technique can work with side-channel power-analysis, which can help you figure out exactly what data a program is using by looking at how much electricity it used to run.
And the best part? Instead of managing all my finance apps (bank, credit card, retirement, mortgage, etc), I just uninstalled them and go to the mobile site. It's not down for maintenance like my checking account app. It doesn't yell at me to update it like my credit card app. It doesn't prevent screenshots. They remember my login honestly more than the apps do.
There is a growing trends towards requiring an app for login to these systems, unfortunately (sometimes it's the only interface, sometimes you can log into a web interface but you need a code from the app to do so).
You can recover information about bits of keys, lowering the security of an implementation of a crypro algorithm, although in principle you could use it to recover other
and others.
Hippocampal sparse coding/orthoganalization/pattern separation and completion/etc may be interesting search terms for you, if you are unfamiliar. If you do happen to be a hippocampal nut like I used to be during grad school, I salute you
Well, it's not about capitalism anymore because the West will be damned if they'd let a Chinese AI destroy the world first, and so too would the Chinese be damned if they'd let the West's AI have that sole honor. ...and so everyone must stay in the race. Solving that geopolitical conundrum is not optional.