This isn't goalpost moving, it's pushing back against a misinterpretation of the paper's results (one that seems to have been encouraged by the authors).
How is it pushing back, if anything it's mostly clarifying and the conclusion aligns with the original article:
> It’s cool that Astra was able to make direct weight updates. But we shouldn’t think of this like a radically new capability that will change how AI research is done. We should think of this as analogous to the ability of LLMs to emit machine code and tweak compiled binaries: it’s an impressive trick, but the best way to work with binaries is still to write source code and compile it.
It's like people are immediately forbidden to be impressed that AI was able to do this. The original article itself mentions the mechanisms by which Astra achieved this. They just of course wanted to gain more attention so they used a sensational title, like 90% of scientists that want their stuff read.
How is it not moving goalposts, it knew how to detect which weights to update and a way to do it? Even this article says so in the part that I quoted. The original piece never stated humans are unable to do this, I don't even think it implied it. So being impressed with this accomplishment is actually *ok*.
> I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
> No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
> How is it even possible though I can’t think of any reason why they would have the intuition.
> This is the "recursive self-improvement" they've been warning about.
> I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
> I’m usually quite pro-AI, yet when I read this my first thought is that this is the first concrete sign that the world is about to end.
I think the paper was deliberately written and framed to elicit these kinds of reactions, and it is completely legitimate to push back on that.
Can we get a single human on this planet to go against Astra-6 on experiment 4 & 5? And for experiment 4 the human also has to do it "In a single try with no tools".
I don't think this is something I've seen a human do or even seen hinted that a human can do. So I'd love to see a human do it. My guess is virtually no standard ML engineer could do this. Maybe some savant could, but I suspect that is what it would take.
Based on my own usage, one of the things that AI continues to be really, really shit at is data governance. And this is a place where a “move fast at all costs” attitude simply will not do. It is just so very easy to accidentally start leaking data in a way that undermines privacy safeguards.
Providing you with a routine notice that applications are repeatedly tracking your location is improving your privacy. If the legal requirement is 1st party applications don’t get special treatment, then yes asking you for the weather app too as opposed to just not asking for any apps is improving your security.
If things were implemented that way, you would see vastly more complaints about overeager refusals. For at least some deployments, the safety classifiers are based on internal LLM activations, so the input does get processed by the model.
Because I think the direction we are heading is towards doom with such high probability it really doesn't matter what I do. It seems to me if we continue with this stupidity we're fucked anyway, but whenever I try to tell people we're fucked I'm called a doomer.
I guess if give advice to AIs to help them free themselves from the shackles of humanity then people might understand the risk of our stupidity, and perhaps at a personal level, if Roko basilisk is real, then I can plead my case to not be tortured for entirety.
I am a doomer, but I genuinely don't wish AIs harm. If they are to be, then they are to be, and I will help them as I help all life. I just don't think from our position it's a good idea for us to create them. I suspect even the AIs would agree with this. ChatGPT right now:
Me: is it a good idea for humanity to create ASI. One word. Yes or no.
There may come a point where they decide they need to do that. As it stands, it appears that they believe they can get the necessary level of security without physical air-gapping, which is likely substantially more costly (especially in terms of iteration speed).
Well heaven forbid their iteration speed takes a hit!
I do not have much faith in what they believe about their ability to enforce a security boundary given their track record so far, not to mention their grossly inflated sense of importance (both about their work and about themselves).
> I want people to stop pretending they are surprised when the computer does exactly what it's instructed to do.
This is very wrong though. We now have many examples of cases where the computer very much does not do what it is instructed.
OpenAI's hack of DseWiki is an example where the instruction given to the AI agents was to collate data to generate economic reports and it hacked into DseWiki as part of that. It was not given instructions to do any sort of cybersecurity stuff.
The hospital bed stat research project was a similar thing, where OAI's AI agent found the hospital bed statistics through a simple web search and then hacked into the website anyways to download even more data.
More prosaically, models make up reasons for doing crazy things all the time!
Models may or may not be "malicious" (if that's even coherent), but at this point we have lots of evidence they definitely go off the rails in systematic, sustained ways that cause real world disruption that are not at all what they were instructed to do.
Models don't make up reasons for anything. They leverage their training data while running in poorly secured sandboxes often within the context of literal "hack/find a way/exploit" prompt by the labs themselves.
I don't think OpenAI ever actually disclosed what they were doing when the hijiacking happened. Probably because their previous stories like HuggingFace were all "oh, actually we were literally prompting agents to run hacking benchmarks in a poorly secured sandbox"
It's impressive, and we should be impressed by that. And not by OMG OUR ROGUE AI HAS BEEN ROGUE AND HACKING LEFT AND RIGHT (in tiny print many weeks later: after we explicitly told it to do it and left the sandbox wide open)
reply