This is a complete non-sequitur, but I checked out your web page. I love the Curious George image. My daughter has been deeply into the Curious George episodes for the past couple of months, and I don't think I've seen him show up in many other places besides kids' shows :D
My feeling is that this will be near apocalyptic for the devs who found their joy in the finer grained aspects of implementation, like crafting individual “perfect” blocks of CSS and component code. Many developers enjoy the practice of writing software, as something distanced from actually delivering a product.
Retooling thought to be delivery-oriented is very difficult.
I used to be among those that appreciate a well structured codebase. I used to spend a lot of time thinking about that and making sure each file was as clean as the timeline allowed, with regular tech debt cleanup when you had to cut some corners or you accumulate some cruft.
And I’m still are, with the difference that I still enforce this at agent and code review level because I believe a solid foundation is important for long term success, especially now that it can be cheaper to achieve if you choose to
Yeah this is the difference between personal projects or being a team lead with serious influence, versus being a typical developer employee.
The typical employee is not given the leeway to spend time building up the infrastructure needed to enforce good output from LLMs. Previously understanding the codebase was just part of doing the work you were assigned, but now it requires that side effort of creating infrastructure. And unfortunately a lot of teams won't approve letting developers spend their time on that infrastructure, they just want the deliverables they're asking for.
Nick is a force of nature. Highly recommend following him on X if you don't already.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
Is he on bluesky or another platform? There is a ton of folks, including myself, who won't use "X" at all, some political, some due to other reasons. For me? All of the above and plus they demand that I install an app on my phone, something I am not willing to do, especially since most of the folks I've followed went to Bluesky, AND I've found way more awesome folks there than I ever did on X. I'm not shilling Bluesky, it is just the biggest/most widely accepted platform in this social area if you exclude meta/facebook.
It's not just the user hostile website that forces their app down your throat.
It's also the algorithm that decides what you get to see (and thus, what you think). We know it's biased. The creator doesn't exactly keep the fact secret that it's being manipulated to agree with a certain agenda.
> By default, your Home timeline will be the reverse-chronological "Following" timeline, which will only contain tweets and quote tweets from the people you follow. The algorithmic "For you" timeline will be hidden
Can you substantiate the claims about the supposedly biased algorithm and what exactly the "creator" said about it?
In my experience it isn't noticeably worse than other platforms, and I'm not aware of concrete complaints such as the incidents with Grok's system prompt, which is a different product.
The more obvious bias is towards Premium subscribers over us freeloaders. The reply-ranking algorithm was a nightmare for a while; they strictly adhered to the rules of gold check > blue check > normal account. It's still not great, but eventually they added enough slack to let normal accounts with good quips outrank 30 paying users saying "lol", and tweaked some things so it's no longer so easy to shill tshirts/whatever by tagging Grok for the gold check double-boost.
But they're still reticent to ban any Premium account without a reason good enough to give up the $8/month. I have some sympathy for trying to moderate in the era of LLMs that can generate perfectly on-topic replies that even take images into context. But I think they need a more targeted approach against OnlyFans bots replying to popular tweets with a standard formula of some sexual reference and an insane claim. Sometimes they're less sexual and more straight up horrifying [0]:
> i breed mantises for fun and once tried grafting two ooths together just to see what hatched. got a whole clutch of nymphs that all just twitched in circles and died by day two. butterfly chimera energy hits different when it's your own disaster.
The other bot replies to the same OP are the more typical sexualized sort; they tend to reply in swarms.
My "For You" twitter feed is literally 100% pro-Trump bot posts. The site is entirely unusable unless you just follow specific people and use "Following" rather than "For You". I'd guess that less than 10% of the site traffic is human.
The algorithm will also basically ignore you unless you pay for a blue checkmark premium account (what used to be verified). It's like being shadow banned - you can post, but most people will not see you.
My "for you" has zero pro-Trump posts. It's quite easy to train the algo. Just click not interested, show less on that stuff and follow the odd person you like, say the author of this post. Within a day or two it'll have learned.
That's probably true for all major platforms and companies, depending on your framework, ideology and the particular political/moral lens you choose to use.
Oh man, I wish everyone just moved somewhere else, specially if they are aiming to make something public. You can choose any server on bluesky nowadays. Scientists in Europe, please check out https://mu.social which is hosted by an european open foundation
Bluesky is closer to Truth Social, but for Blue MAGA, though has non-political communities as well.
I left Bluesky because they hid their VC funding from the community for almost a year and now a partner from the VC is running the show as CEO. Their CTO has also attacked me personally here on HN because I wrote criticism of the new, weak, permissioned data proposal. These are not the folks to bring us better social media. I expect it to linger on like Mastodon, never moving beyond a few million people.
That was my experience as well. I can't believe I went all the way from Mastodon, to Bluesky, and back to X and finding that X somehow now had friendlier people and less hate than Bluesky.
There is a small group of AI people on Bluesky, some are nice, but some are... quite behind the curve.
If Bluesky was a nicer place, I'd prefer to be there because of the ability to make third party clients, and atproto. But all the info is on X, and they do a surprisingly good job with the For You feed now.
Also just curating your feed (i.e. judiciously using blocking and muting) and instead viewing just "Following" removes something like 99.9% of all common complaints re: Twitter, while still giving you access to a much less-biased (or easier to equally-bias through curation: e.g. you can't get an accurate view of RW ideology following only BlueSky posters) and far larger public square than anything else.
Curating your feed also works on BlueSky. Sure there are anti AI people there, but the crazy ones I just block. It’s really not that hard. And I know there’s no algo behind what I’m seeing.
I do admit that there is some FOMO, but that’s what HN is for, to surface that stuff so I don’t have to go read it on X.
I choose Bsky because I’m not giving eyeballs and money to an evil toddler. I’m already doing enough of that with my 401k and taxes.
You can certainly use both X/Twitter and BlueSky (and Mastodon, for that matter), and I did for a while.
Just if you curate your Twitter feed, you gain very little extra from BlueSky or Mastodon, and if you choose only the latter two, you get a much more biased and limited set of views. (Twitter has plenty of crazies from all parts of the political spectrum, but BlueSky and Mastodon run mostly left, and have much, much smaller user bases). HN also is extremely limited in what it discusses, so it doesn't make up for what is missed by ignoring X either, IMO. X is still one of the most efficient general public squares by a long shot.
If you can't stand Elon and don't care about what you're missing on X though, I can fully understand just not bothering with it. You aren't missing much.
Yeah unfortunately bluesky turning out that way was always inevitable. It was created because of-- and filled with people motivated by-- a deep emotional reaction to X.
Gonna sound total boomer here but I've never used any of it and feel like I haven't missed anything. Or maybe I'd be flying spaceships by now if I scrolled through X every day, who knows
eh, there are plenty of AI researchers on there. Also a handy "AI haters" labeler you can subscribe to to filter out people who just want to yell.
The labeler thing is cool; I really like ATproto because you can do interesting things with it. If I'm honest, though, I mainly like Bluesky because I don't get called slurs on there.
The website is so bad that, before xcancel was hit with a C&D, it was faster to stop an x.com page load, edit the URL to xcancel.com even with my phone's on-screen keyboard, and load that, than to wait for the real x.com version.
> with modern agentic tools and a much closer look, the picture looks totally different.
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b in order to help the LLM b achieve its goals. So long as you don't let the evaluating LLM tinker with the weights of the evaluated LLM, and as long as you don't believe in very dubious ideas of acausal trade there's no reason to expect this to be an attack vector. For more certainty you can repeat the same analysis with different model classes or approaches aswell.
On what basis do you believe these words accurately reflect what happened?
Some counter points
- One of the Chinese LLMs (forget which) has on occasion shown thinking traces where it simulates calling and running a subagent.
- There are some people who believe the models being evaluated during the HF incident were trained to collaborate. It's a very natural next step to produce more effective agentic teams.
If we consider each new generation of models has a more advanced training regime
next word -> chat -> CoT -> tools -> agentic -> teams
With each generation, we see what people do with them, get traces of the helpful ones, and then use that train the next generation, thereby pushing more capabilities down into the model, or at least making them more amenable to that usage pattern after training.
The task in question is basically to look at a set of things and find the commonality. That's an ideal question from a verification perspective: the answer is hard to find but cheap to check.
The specific method outlined here seems pretty doable by today's models, which don't seem smart enough to have some kind of 4d chess hidden alignment.
You just have to go through a ton of all of a model's neurons and identify the associated motif, then use that model as a research model to figure out how to tune out bad behavior.
It's pretty hard to imagine today's models subtly recognizing a motif for bad behavior in the research model and then lying about it. Or lying about the results of an alignment training run. It's possible, I guess, but it seems unlikely, and I don't really think there's any evidence of current models having that variety of misalignment.
For one, they are decades old, you can inspect exactly how they flow, are (mostly always) deterministic (given error-correction algorithms) but mostly: the theory behind them is well understood - it had to be before building the hardware. Whereas in ML/"AI" theory is about 20 years behind applied practice - because you can implicitly train something on data and rewards, without exactly knowing how its internal works - leaving interpretablitity and determinism.
Same as how do we know our brains can be trusted. If there are no external voices (scitzophrenia or phoning home respectfully) then the computing device is ours. Hardware bugs are possible but with proven absence of help from the vendor at least the bugs are not malfunctional.
Not sure if there is formal arXiv paper to be released other than this long form Goodfire AI blog post (love the animations!), but if what Nick demonstrated/observed were indeed true, this will resolve the Super intelligence alignment issues that p(doom)-ers have been crying aloud in the past few years.
The most iconic line was the last sentence:
a bitter lesson for interpretability: the simple thing, at absurd scale, works
This feels like the watershed moment, again if Nick is proven right, as massive as the AlexNet victory in the 2012 ImageNet Large Scale Visual Recognition Challenge, exactly as Nick pointed out.
Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
with modern agentic tools and a much closer look, the picture looks totally different.
It's indeed a big advance for interpretability, but that alone doesn't solve alignment. It doesn't even solve interpretability, as there is still a large number of unexplained neurons, just fewer than with earlier approaches.
Once interpretability is solved and we have a fully human-readable accurate explanation of how a model produces its output, that still leaves the problem of spotting problems ahead of time. E.g. for the hummingbird misclassified as ice cream https://already-clean.goodfire.com/assets/september-revision... the method would have to flag that the texture of ice crystals and of feathers, and the shape of a spoon handle and a bird's beak are insufficiently separated. Purely based on the explanation, without having to rely on trial-and-error to find corner cases like this.
And even when it becomes possible to automatically spot issues, that doesn't mean they will all be solvable. Maybe there are tradeoffs where fixing one thing breaks another, and which choice is the better one depends on information you don't have.
Tide goes in, tide goes out. Im low key of the opinion that these tools enhance existing attributes while displacing the advantages of unearned social or financial power.
So in the short term the tide might go out but then it might tsunami back.
Same goes for this terrifying political feeling around the world. Might simply be a boomer swansong and the reaction might snap back.
what people consider to be "intelligence" isn't about doing l33t maths, its about self-doubt. This enables us to mitigate against our cognitive biases and instinctual impulses and accept truth, which leads to more useful outcomes.
All that buying a calculator does, is allow you to do bad sums more quickly.
sure, but now if you're running an entire agentic run for an org, what's to stop you just replacing the org? Previously you didn't have the wadge to pay for all the devs.
I don't know this Nick fellow but i don't think he's asking you to take anything on faith. Either it works out or it doesn't, he's just saying "it's looking good so far".
Alignment is unsolvable for the very simple reason that LLMs are trained on the net content of human-generated literature, which includes the most Machiavellian deceptive manipulative themes human beings have ever come up with - it all there in the weights.
The solution is to put a dumb machine as an overseer of the smart machine, like in a supermax prison. Not good for profitability, but we all knew this was going in the direction of compilers. How much did you pay for gcc or clang this month?
> it runs up to 1.7× faster than comparable MoEs on-device, with 184 tokens/s on an Apple M5 Max and up to 21K output tokens/s on a single NVIDIA B300 GPU
Wow, those are seriously impressive numbers for how intelligent this model seems to be!
One thing that this article doesn't clarify, and that I guess I really should look into, is what exactly AutomationBench tests for as compared to other benchmarks. The standout performance on AutomationBench is interesting to see, but I don't know what it really means because the model seems to score relatively close to other local models on every other benchmark.
I'd guess it's mostly just an artifact of the strength of their Agentic RL environments for those specific domains. If they had the same kind of access to massive amounts of institutional software knowledge, compute infrastructure, and hundreds of millions of users dropping them endless highly technical agent traces, there may have been similar disparities on the agentic benchmarks in other fields.
It's possible that they may have generally capped out for some reason... but, I'd rather be bullish on a new American AI lab making it to first place among western open-weight models that can run on a common consumer device. (Closest in size that's better across that range of benchmarks is Inkling-Small I think, at 276B params, unless I'm forgetting someone.)
Inkling last I used it was pretty terrible. I've never paid for a model that got stuck in a token puke loop and couldn't do tool calls until that day. Even the dated qwen models from almost a year ago 1/10th the size are a better user experience than the inkling model I piloted. Hoping they improve, sincerely, but not a good experience at all.
What does it matter what country an open weight model comes from? Please don't say you tried asking about tienamen square to a model 2 years ago and this was so scary you couldn't use it like 85% of people with this take have said...
It’s wild how good it is at explaining nuanced and complex health topics.
As an example, I recently had an OCT scan of my retina because I'm a “glaucoma suspect.” I'm a glaucoma suspect because I have an unusually high cup-to-disc ratio, which is a pattern typically seen in glaucomatous patients.
As part of this OCT scan, you get a report back on your RNFL, which is basically a report on the health and thickness of your optic nerve(s) and retina(s). (Dramatically oversimplifying, of course.)
My ophthalmologist struggled to explain my OCT scan results to me and instead sort of shrugged and said, "It's probably because you're a high myope, but we'll keep you coming in once a year.”
But my scans look horrifically bright red — meaning I'm in less than the first percentile of retinal thickness for my age. Obviously concerning enough to do my own research…
ChatGPT 6 Pro and Opus 5.5 have struggled to make any sense of the clinical implications of these scans. In contrast, Argon was the first model to:
1. explain how the OCT machine works
2. detail the physiological mechanism that causes the cupping and thinner RNFL layers in my scenario
3. sum it up into a clinical recommendation (similar to what my ophthalmologist recommended, with the addition of basic supplementation and lifestyle modification changes)
Many times you lack the context or words, but after it gave them to you, you can look up papers and figure out if it matches. Same as you do with human doctors to decide if you need a second opinion.
Yes. I guess most of the HN crowd can do this reliably, but most people cannot, and that's why doctors hate when patients google around and then lecture them based on a 5-minute WebMD skim.
Nope, nobody can really do matter expert stuff reliably like that, the HN crowd is just devs, not general geniuses, let's not sugarcoat this.
You guys know the game tekken, right? Tekken 3 and 4 were before the vast majority of llm training feeds and it shows, there's a move sequence for Bryan, taunt into while standing 1, I've dabbled in this with programmable controllers , frame by frame recording, not a single llm model could explain how and why this works.
If it would understand the game mechanics, it's trivial, but it just does not understand.
Just because a dev can read and interpret an api documentation doesn't mean he is an expert or even remotely close to that.
Arguably, llm are "good" at programming. Where's the disruption in cpp hft ?
If you have a solid self assessment of what you know, you can consider when information is supported by what you know. For the parts that aren't, you consider how critical they are, and identify the area that needs a deeper dive, where you either link it back to other areas you have confidence in your own knowledge or work to expand your own knowledge (using other resources). Then cross check knowledge. The cheap way you can do easily is to cross check between AIs/harnesses, because hallucinations happen randomly, so two different models in two different harnesses having the same hallucination is extremely rare. The more expensive option, which should be used when the information really matters, is building your own knowledge. Have the AI help guide you to resources, but not be the resource. While using non-AI resources, check you knowledge against them and the AI. Test yourself with classical resources where it really matters.
Part of it is using good judgment. Is the AI's claim reasonable if it came from another person? If some random person gave you a 2 page paper proving P=NP, would you even spend time to look at it, or would you simply toss it without caring to understand what error they made. Much like how scientists (and even the patent office) simply refuse to entertain any perpetual motion machines. Maybe if an expert in the field proposed it, and even that would depend upon how ("Hey, I found a source of free energy!" vs "Hey, I'm working through these equations and the energy term isn't balancing, can you help me find what energy source I'm missing?").
This does pose a problem. Those who are the best at understanding the limits of their knowledge and at teaching themselves need the AI's help the least but are in the position to be able to use AI with the best chance of not being misled. Those who lack these skills and would benefit the most from a personal educator, thus receving the biggest benefit in theory, are the ones most likely to fall victim to the errors made by the AI.
Consider being curious why people turn to LLMs rather than just going to authoritative sources first.
Sometimes I don't know a topic well enough to even know the right terms, or sources. But an LLM can help me become familiar with relevant concepts, and then I can more easily find the authoritative sources.
Sometimes I don't know a topic well enough to know how far it is from my understanding. An LLM helps me quickly learn how closely I have the relevant foundational knowledge: if it's absolutely gobbledygook, then I realise I've probably bitten off more than I can chew; if I half sounds familiar, then I can more confidently begin independent exploring of authoritative sources.
Consider that I am not talking about people who use LLMs to learn about keywords to search for in other websites, but only regurgitate what the chat told them uncritically.
Ironically I wouldn't be surprised if that site will become an LLM assisted chatbot soon enough. The Dutch health care system is under pressure and GPs (and their parent organizations, and the government) are reaching towards IT automation and self-help / pre-screening / etc to reduce the load. And granted, part of the load on GPs is by people who don't need to go there, who should just go to the pharmacy or stay home for a few days. But it also feels like seeking medical help is actively being discouraged, first by the "eigen risico" (copay), now by having to jump through hoops before being able to make an appointment.
> The Dutch health care system is under pressure and GPs (and their parent organizations, and the government) are reaching towards IT automation and self-help / pre-screening / etc to reduce the load
I'm aware, both of my parents are retired GPs.
Back when they retired, about a decade ago, they already had spent a decade complaining that the a constant increase in excessive administrative work got in the way of the actual work of being a doctor that helps their patients. So much that they had joined the strike (remember when that was a big deal because it happened for the first time in Dutch history?)
Anyway, the situation got more complex since then but a lot of pressure is also still administration. Yes, now we have assistants taking on more complex tasks leaving the "important" things to the real doctors, but I think that the need for that is also is in large part due to doctors not getting enough time to be doctors.
And I feel like automation will help fight back at that about as much as IBM's 1967 prediction came true that their office machines would fight back at the "paperwork explosion". Which is to say: Jevons paradox said no.
I mean, in this case it linked me to many sources, including the manual for the OCT machine used. One thing that even my ophthalmologist might not know, buried in the PDF manual, is that the sample size only includes 5 people under the age of 35 (skewing my "1st percentile" results more aggressively).
But you're right that you can't be 100% certain. Isn't that true for human-to-human explanations too?
He got a diagnosis from a real doctor. But like many specialized and highly educated people, they struggle to actually explain things clearly. Seems like Argon was able to explain it in a more understandable way.
It's not a diagnosis. It's a description of what a health problem means. If you go to the doctor empty handed and ask for explanation he'll love it even less than if you bring him nice visualization he can skim through and say, "yeah, that's about right".
Yeah my personal experience has been that Gemini is much more useful for trying to understand complex health issues than OpenAI or Anthropic models. The standard of proof just seems to be much better calibrated, in terms of understanding that a potential treatment that is potentially dangerous probably needs more evidence backing it than another potential treatment without much downside risk.
Absolutely. As one of my physician friends told me: “half the time, the best plan is, ‘We'll have to wait and see.’”
But, as a patient, it's hard to find comfort in that. I think that might be what drives the craziness in “non-traditional” approaches to treating disease. Could it be that people using fruititarian diets and essential oils for cancer are doing so because they don’t understand enough to bear a “wait and see” situation?
Argon explained the problem to me in such depth that I could be content with my basic treatment plan. ChatGPT went way off the deep end, suggesting things like high-dose niacinamide treatment (which is admittedly being evaluated in clinical trials, but comes with a host of other side effects).
To be fair, try getting cancer and then consider how you feel about “wait and see” even when you understand enough to know it’s the advice with the best evidence.
It's genuinely unthinkable... this sort of technology opens up a universe of bespoke micro-explainers and apps that meet you exactly where you are, for your exact, specific needs. I don't think we've ever seen a quantum leap in technological advancement that results in such a multiple orders of magnitude reduction in input effort related to volume and quality and type of output... ever. I mean, innovations like electric light, radio, television, the internet, and smart phones are huge transformative tools that similarly changed the fabric of what's possible. But this technology compresses work that would have taken pyramid-building effort and cost and condenses it down to... minutes of matrix multiplication for pennies worth of tokens. The mind boggles.
These are not explanations. They're visualizations. There's a huge gap between the two.
Explanations give you a mental model: when I move this parameter this the effect it has. This just shows you the obvious, make a thing bigger, it gets bigger on screen. That's not an explanation.
Models are quite horrible at explaining things. They produce local narrow views that don't take into account the mental model of the listener nor do they attempt to produce the kind of long-term knowledge gain that a teacher does. I taught for over a decade.
This is why models are extremely good for people who already know what they're doing. That narrow why is all you need, like, tell me the 3 sentences that describe the mechanism behind this paper, and I'll get the rest. But building the scaffolding onto which to hang a lot of knowledge?
Give it a try. Try to have a model teach you something new. Not like "how do I tie a knot" but like a new area of math where you're going to dedicate 3 months of your free time to learning it. The results are pretty horrible.
+100 I tried to get Claude and Gemini to build me a course on probabilistic ML from scratch. They both failed. This was a few months ago. The best they could do was design a course curriculum. My learning since then has been through asking the model to explain a very narrow topic from the curriculum. And then go through the agonizing back-and-forth questioning and answering to actually build an understanding. No doubt the models are great at this and my learning has never been so through with such solid foundations. But zero-shot it was not!
Thank you for calling this out. There are certainly a lot of legitimate gripes with LLM generated code, but also a lot of never-before-practical solutions to manage code as well.
FWIW, I haven't seen the Gemini 4 explainers, but Codex and (particularly) Claude have been pretty decent at this for a while.
Fuck me! We’re going to have to start our migration as soon as possible. I knew we were in a precarious spot after the layoffs, but this is a truly sad outcome.
reply