Not making any claims about a specific model's capabilities, but all these "can interpret model weights" posts brings to mind the line in Shakespeare's Henry IV
GLENDOWE: I can call spirits from the vasty deep.
HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?
I can interpret weights too, but will that interpretation be correct?
These are famous games tested here though, and I tried to feed one problem to DeepSeek 4.1 Flash 1017 and it comes back at 66% confidence it is a Pac-Man (incorrect). That smells to me the problem setup heavily hint towards these 2D games. Might be resolution related to be honest.
I tried the same experiment with DeepSeek 4 Flash as well, on more than 4 games. And I think you might be right, the performance is much worse for other games:
https://news.ycombinator.com/submitted?id=wohoef
Great post. I looked into the reasoning trace, and it seems DeepSeek 4.1 Flash 1017 does know all the games you listed in the post, so there is no knowledge gap. Not sure about Qwen though. I will not be surprised if it doesn't know all these games.
From what I saw in the paper, no they did not. They tested that the specific change introduced increased performance on a single task (the one mentioned in the prompt). They did not study the impact of that change on overall model performance across a variety of benchmarks. But maybe I missed something.
Further than that, aren't "weights" just another sequence of tokens? We see them as vectors, pass to the API or as strings into a chatbox interface. Is it so surprising that the high-dimensionality linear algebra machine crunches them and gives us back useful vectors that can be mapped to text?
Isn't that what this whole technology paradigm is about? How is it surprising?
Granted, it is cool. But I see so much of "look at this cool thing I did with LLMs" and honestly... we need more structured text, more explicit documentation, more ergonomic and intuitive CRUD applications.
The amount of "look at cool things" is growing way faster than before and they are all sort of low effort now because the "agentic" LLM-VM looping apparatus can do a lot of heavy lifting and run with our prompts.
And I mean, if one has jargon, knowledge, and intention, they can also leverage that into great structured text, ergonomic and intuitive CRUD apps etc, all of which add great value to our mundane pursuits of applied research, software products and services and so on.
Honestly, I begin to think the problem is people dazzling themselves instead of doing the housekeeping that is the bulk of life. Technology choice is entirely secondary.
Idk. Yes of course it can interpret weights as a bunch of comparatively low dimensionality tokens but the way the weights operate as the model gives them incredibly high dimensionality, for very large models potentially a downright absurd number of pathways and influence on the overall result.
But frontier AI can work out math and of course this is all just...math. So I imagine given enough time + testing and probing that it could work out what each weight does/influences and trace all the pathways.
And presumably this means that it would be possible to train a model to interpret weights as current models interpret human language although our language remains pretty constant, even different quants of the same model will experience enough change to mean no real basis for training a model to do that.
But I suppose it would surely be possible to train a model to interpret the weights in a generic "these are weights for an MLP" way? But then if I give a partial sentence from the middle of a book, you can pretty effectively guess what's going on locally but perhaps not globally. If I give you a chunk of loosely correlated weights from the center of a model...I feel like that's much, much more difficult but that the same applies?
I understand linear algebra. It seems like he probably does, too.
Are you sure you have read and understood his point?
It seem quite possible that the flattened-model-weights ingested into a model have a much greater fidelity of representation in a similar model's KV cache than they do in a human's working memory, and that being able to loop and digest and chunk makes it unsurprising that models can excel at this type of task.
I think that's the point the person is making. Do you disagree? If so, why?
> Bringing up the KV cache is a sign you don't understand what you are talking about here. The KV cache is a computational efficiency thing and has no impact on how these models actually work (and isn't needed at all). The vague analogy to human working memory also just takes the cake here.
You seem to really like to tell people that they don't know what they're talking about more than you seem to be parsing what they're actually saying.
I remember when I used to look for reasons to pounce on people for not knowing what they were talking about based on a superficial thing or two rather than try to understand what they were saying. I mostly got past it as a teenager, though.
> I can interpret weights too, but will that interpretation be correct?
Chances are much higher than a model can figure out what the patterns are, decipher them, and retain that knowledge in a meaningful way it can reason about. Me and you might take years or months to do all of that on our own, with a "map" of the weight we might be able to discuss it, but not as effortlessly as an LLM.
Honestly, I had told a few different colleagues that reading a model is probably not that exclusive to frontier models. It's probably yet another reason OpenAI / Anthropic don't release any big open weight models, but thankfully Google and Microsoft release reasonably sized models. Imagine a new GPT open weight release, it would get dissected drastically.
Thinking about it, it makes a lot of sense that LLMs could easily be trained to interpret weights. The attention mechanism of an LLM is very well suited to analyze a set of weights (in text form), since the attention to each weight in the text via key/query can sort of emulate the connection structure of the network that the weights represent.
Biblically speaking, spirits from the vasty deep must be demons since they are from below and angels, God and Jesus are from above. So in that sense it’s not very impressive. Demons are something a sane person would rather avoid.
I mean, if you want to inject religion into this, that interpretation that it "must be demons" isn't even supported by the source text.
In 1 Samuel, King Saul visits a medium in Endor in order to speak to the (dead) prophet Samuel. Human dead go to Shoel (in familiar terms, it is similar to the Greek underworld) in the Old Testament. Later on in the New Testament, the idea of a really really bad place is introduced in the form of Tartarus, which is specifically where the rebelling angels were sent. But that is not the same place human dead go.
The idea that everything "not Heaven" must be Hell is an invention, caused mostly by dogma that rose from bad translations of the Greek and Hebrew in the middle ages (for example, King James - itself mostly correcting minor errors of the older Bishop's Bible - refers to all of it as Hell even though the original text most inarguably refers to different places different beings go).
I see your point about Sheol not being Gehenna not being "the bosom of Abraham" not being the New Jerusalem and so on, and you might already know everything I'm about to say, but just to clarify for others' sake: I think it's wrong to say that the "really really bad place" is uniquely introduced as Tartarus (AFAIK mentioned only once in the NT, in 2 Peter 2:4[0]).
Jesus clearly refers to a very unpleasant place in e.g. Mark 9:47–48[1], Luke 16:24,[2] and Matthew 25:41[3] (where it's explicitly stated that false prophets will go to the same place as "the devil and his angels"). Other references to an unambiguously bad place, as the sole and final destination of unbelievers, are throughout the New Testament, e.g. the "lake of fire" in Revelation 20:15[4] and 21:8[5]. Revelation 20:14[6] even says that "Hades" will be thrown into that same "lake of fire" (which of course supports what you're saying about Hades being distinct from Hell).
As a final note, even the reference to Tartarus in 2 Peter 2:4 speaks of it as a temporary holding place where "angels who sinned" are being "kept for judgment", implying that there's a more final fate awaiting them (identified in Revelation 20:10[7], at least for Satan, as the aforementioned "lake of fire").
The main point I'm trying to make is that, while it's certainly true there are other "places where dead people/angels go" than Heaven and Hell, that's only true at this point in time. To my knowledge, the New Testament is consistent about a final judgment where there are only two possible fates (the New Jerusalem, i.e. Heaven, and the lake of fire, i.e. Hell).
[0] "For if God did not spare angels who sinned, but cast them into the pit [Tartarus] and delivered them to chains of darkness, being kept for judgment;"
[1] "And if your eye causes you to stumble, gouge it out; it is better for you to enter the kingdom of God with one eye, than, having two eyes, to be cast into hell [Gehenna], where their worm does not die, and the fire is not quenched."
[2] "And he cried out and said, ‘Father Abraham, have mercy on me, and send Lazarus so that he may dip the tip of his finger in water and cool off my tongue, for I am in agony in this flame.’"
[3] "Then He will also say to those on His left, ‘Depart from Me, accursed ones, into the eternal fire which has been prepared for the devil and his angels;"
[4] "And if anyone’s name was not found written in the book of life, he was thrown into the lake of fire."
[5] "But for the cowardly and unbelieving and abominable and murderers and sexually immoral persons and sorcerers and idolaters and all liars, their part will be in the lake that burns with fire and brimstone, which is the second death."
[6] "Then death and Hades were thrown into the lake of fire. This is the second death, the lake of fire."
[7] "And the devil who deceived them was thrown into the lake of fire and brimstone, where the beast and the false prophet are also, and they will be tormented day and night forever and ever."
the freeway miss is the tell imo. the one obscure game of the four and it just called it pac-man, which looks like pattern matching toward famous games, not reading what the network computes
>I can interpret weights too, but will that interpretation be correct?
In TFA case, given that they were able to guess what the weights did (which the researchers knew in advance and could cross-check) and to even tweak and improve them, the answer is yes.
Indeed. Some of the worst software interfaces I've ever dealt with were fully "dogfooded" and that was the cause; the people writing the software did use it every day but (a) in a very idiosyncratic way and (b) in a specific, very constrained environment.
The devs did use the software, but used it in such a different way from everyone else that when the devs optimized for their own case, it made the software actively worse for everyone else.
Indeed, it's a lesson I learnt the hard way and is much like proofreading something you wrote, you don't see the errors as easily as someone else you asked to proofread it usually does (well until you hit Send then you usually see all of them instantly).
A tiny thing about Asimov's laws of robotics is, most of his stories involve cases where they don't actually work.
Spoilers for a 73 year old novel, but for instance the plot of Caves of Steel is centered on a robot with a perfectly functional 1st law abetting a murder.
"Runaround" (spoilers, 86 years) involved a robot getting stuck in a loop bouncing between the 2nd and 3rd laws, and a human having to risk their life to unstick the robot.
Et cetera.
What is harm? What is an order? How do you trade off between different kinds of harm, or deal with conflicting orders? The 3 laws are simple to state, but hard to apply consistently in real life.
It's a little less dramatic than that. They did send a demand letter to automatic.css saying that since 'automatic' and 'automattic' differ by only one letter, are pronounced the same, and are marketed to similar groups the name is likely to cause confusion, hence automatic.css should change its name.
Which is not the same thing as claiming the word automatic generally.
Personally it sounds like a loser if they took it to court since (a) automatic is a common word and (b) they already asked automatic.css to add a "not affiliated with word press" disclaimer (which automatic.css did add). IANAL.
Directly telling someone that you have a problem with something they did and why you think it's a problem is not "passive aggressive" by any stretch of the definition.
GLENDOWE: I can call spirits from the vasty deep.
HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?
I can interpret weights too, but will that interpretation be correct?
reply