Math research isn't all or nothing on a single problem. Rather, the problem would serve as inspiration to explore achievable subproblems, weaker problems, related topics, etc. These would be interesting papers by themselves, but if lucky may also lead to progress on the main problem. So this leaves lots of room for using AI as an assistant.
Why aren't they running agents on isolated computers without Internet access? This way, breaking free of containers would not matter.
It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.
Sooner or later, agents must be connected to the real world to do economically useful work. The sandbox failures are bad, sure, but the underlying misbehavior is the root problem here.
I think the way they're connected to the real world doesn't have to be as free-form as we're doing now.
If you think AI can help design new drugs, great! Let's pick some restricted DSL for describing molecules, preparation instructions, hook it up to the right set of lab robots that can pipette and centrifuge and whatever, _restrict its output to that DSL_, and let it iterate. The 'let an AI design drugs' process doesn't require that the agent can also, say, hack a niche German wiki.
For every area where we think these AIs can be _so valuable_ because they'll find solutions that humans wouldn't (or find them faster), then doesn't that value also imply it would be worth it for some humans to do some advanced setup of their specific domain-specific tools so it can be run in a sandbox equipped with only what it needs and not more?
Anthropic claims that their intention with the agents that led to the Hugging Face hack was an offline experiment running against a hacking benchmark.
Either you're being pedantic and missing the entire point, or you're saying that Anthropic lied and made a fake sandbox knowing their agents would need to connect to the internet anyways.
But the specifics here are the thing I was describing. This was cyber capabilities training on model(s) that were in a relatively unknown state of alignment training.
Because of the unknown alignment (and for varied practical reasons I guess) the training is intended to be inside a sandbox.
Because the training is on cyber capabilities, the models need access to simulated cyber environments, including target endpoints, including package managers, etc.
For package management, they set up Artifactory as a secure proxy. Agents ask Artifactory for packages inside the local network, and Artifactory serves them directly or goes to the internet to fetch if they are not cached. But the agents hacked Artifactory to steal its internet access.
So: to train cyber abilities, you need to at least approximate cyber environments. To realistically approximate cyber environments, you need to either pull a full copy of the entire internet to local or to use proxies. The former is pretty impractical, and the latter is exposing our limits at creating secure proxies. Yes, any specific failure can be mitigated, but the models get stronger and stronger. Fingers-crossed this is not escapable doesn't feel great!
How about they create a virus that copies itself onto USB drives? Getting that virus out there doesn't sound that hard, just creare some crap game.exe and put it on itch.io for free. All it takes is for a kid to borrow nuclear-engineer-dad's USB drive and plug it in their infected PC.
I'm not a fan of using AI, it doesn't amplify the work I do that much, but I'm terrified by what is already possible today.
> All it takes is for a kid to borrow nuclear-engineer-dad's USB drive
Do nuclear-engineer-dads take work usb drive home. Are their computer even allowed to have usb. I'm typing this on a dell latitude, and you can disable usb and other peripherals inside the bios.
My last job, I had to apply for an exemption to use the USB ports. And it was run of the mill software engineering.
I think there’s a 2022 way of thinking how models are trained/evaluated, and there’s 2026 way of doing things. Models that are not evaluated with public internet access are pretty useless nowadays for daily operations.
...why not pitch a start up that sells AI agent that don't train with/for internet access/usage?
1) Because effectively all of the hardware used for that task is earmarked for the major LLM manufacturers.
2) Because in a just world, the major LLM manufacturers would get punished for their flagrant deception, lawbreaking, negligence, and recklessness, [0] go out of business, you'd get all the hardware that they were using at fire sale prices and save a ton of money compared to attempting to start up now.
[0] Based on their recent claims, building WMDs [1] without adequate safeguards is the most negligent and reckless thing they've been doing.
[1] It's fair to call anything with a 10% chance of wiping out all of humanity a WMD.
I feel like you missed my point. To be slightly more explicit, I find it comical how willing security people are to build a "secure" product that's completely useless for getting work done.
Because then it wouldn't scare the governments into regulating models that are undercutting them. This is just an obvious tactic too and it seems like it might be backfiring a bit. Congress might be a bit dumb, but they're not that dumb.
Your second clause is sufficient there: they're not going to have a good time trying to sell AI that can't access the internet, because accessing the internet (and other computer systems) makes AI much more useful.
Motivation can come from many sources, not only grading, e.g. individual encouragement, exciting applications, day-to-day relevance, interest shown by peers. Also students can just be intrinsically motivated.
It may come in different styles. But what do you do with students who don’t have motivation beyond grading/punishment? Some people just do not have motivation unless somebody standing by them with a stick.
According to the article, #1 leads to "failure and drop-out rates of 50 to 80%". #2 and #3 might work for organized and motivated students, but graded assignments from other classes will be prioritized by most.
Given the education literally does not matter anymore for white collar work, I'm inclined to agree with you.
I'd rather a MUCH larger amount of professors and teachers come to class to find their classes deserted (by their own hand), and indeed, their own justification for existing gone. The world would be a lot better off. Gatekeepers are anti-human.
Then their parents come in and complain about where the $30k that semester went. This is a consequence of the exponential rise in tuition. You're now in lawsuit territory if you don't deliver the goods.
A major goal of experienced programmers is to manage risk, often by reducing unnecessary complexity. LLMs seem to be the exact opposite of this, perhaps soon producing volumes of code noone has time to review.
Should the industry go this way, there very possibly will be some reckoning later with a misaligned model.
That said, there are certainly less risky ways to use LLMs positively in development!
But when I think about it, communities like HN don't exactly have only the taste I want either, do they?
That's always the case, really. There are people whose daily lives fall apart because they read too many books, but that usually isn't categorized as a form of addiction. So sometimes I wonder how addiction is actually defined.
This is what happened with me and BASIC. I had to upgrade to C when my programs became limited by complexity, speed, and difficulties robustly integrating audio.
The goal of PhD programs has never been to have every student become a professor. Also, strong STEM PhD students generally don't need much help finding jobs after they graduate. Claiming that PhD students "aren't prepared" for industry careers is simply saying that students lack industry experience (obviously, being students!).
The problem here -- and why universities even do research at all -- is that industry's support of basic science is not very stable. At least when I was in school, industry research generally was more applied and engineering-focused.
