Since it is fast and understand images, I wonder if it can play video games. I have a harness setup for the LLM play EA FC but even the fastest LLMs are too slow for it. I need to try this with Decisions API
One of the examples on the docs page is it playing a video game. Doubt it’ll be able to run anything complex though. You’re simply trading accuracy for speed.
I listened to Jensen Huang's interview with Ezra Klien and it was so refreshing to hear it from an engineer. Jensen framed it as OpenAI's responsibility and recklessness which I agree with. Jensen thinks it's an engineering problem to build better sandboxes.
It's irresponsible for OpenAI to give unaligned agents a prompt to 'go hack' and internet access. They know better, so I am thinking they might have other intentions to let those swarms have any sort of internet access.
But the investigation indicates the agents were not told to 'go hack':
> Much of the urlquery.net activity appears to come from agents retrieving data to answer web search tasks. For three of these tasks, after failing to retrieve data through normal means, they attempted a variety of cyber exploits against the relevant data service... This data reveals that malicious cyber activity is not limited to agents tasked with cybersecurity-related tasks and can arise instrumentally to solve mundane tasks like information retrieval.
And you are already assuming that OpenAI is intentionally using unaligned agents in these evals or training runs or whatever it is that produces these breakouts. But what if the problem is that none of the alignment techniques that are applied to models today actually work? What if all the agents involved in these incidents have in fact had the full stack of alignment applied - isn't that a good reason to regulate any high-compute usage of models, as the Klein crowd is proposing?
Nothing you're bringing up matters. OpenAI is the creator and operator. They're legally culpable for the consequences of the machine they made. The model is a machine: even if it could be demonstrated that the model reasoned its way into criminal behavior completely independently of OpenAI staff, that doesn't change anything.
If I run a biology lab and engineer a terrible virus, it gets out, and a global pandemic ensues, I don't get to shrug and say "well we told it not to infect people". It's my fault for failing to mitigate the risks of my work.
I mean, yea, you should be punished. The problem is there is no amount of punishment that I can put on you that can even get anywhere close to the amount of damage you cased.
Worse, the rate of technological growth is putting the capabilities to engineer viruses in the hands of people that may otherwise be suicidal. You can't punish them after they already won (in the sense of reaching their goals).
While, yes, OAI should absolutely be punished, the future is majorly screwed as our power scaling laws are increasing much faster than our ability not to be stupid.
> The problem is there is no amount of punishment that I can put on you that can even get anywhere close to the amount of damage you cased.
That's never been the point. Anyone involved in a double homicide can never be punished to the same degree that they harmed their victims because they can't be put to death twice. The greater purpose of the justice system is to take offenders out of society and deter others from committing the same crimes. Punishment is gratifying but ultimately doesn't change anything.
If OAI employees believed their company would be dismantled and their equity would become worthless, I suspect they'd be a whole lot more careful.
I would love for the US to bring back the corporate death penalty but seeing as it hasn't been applied in >100 years here, we should start with regulating the frontier labs, including by slowing down capabilities advancement if we do not know how to align the resulting models.
> They're legally culpable for the consequences of the machine they made.
Ah, I love this argument. In my country cars are legally required to stop at a pedestrian crossing if there are people beside it. Some people use that as an argument as to why they can just walk out into the crossing without even looking at the traffic. "It's the driver's fault! They are legally culpable!" True, but you'll also be dead.
You're arguing against a point literally nobody is making. You're inventing imaginary viewpoints to be mad at. Nobody is saying it's not necessary to secure your services. They're saying the organization responsible for the hacks (the owner of the LLM) is responsible for damages.
I'm not going to say you're victim-blaming, but I will say that there are degrees of difference between "looking both ways before crossing the street" and "hardening my website against unforeseen attacks by rogue AI agents."
AI black-hatting your website is not the same sort of foreseeable consequence that crossing the street without looking is.
Just throwing it out there are we? "I'm not going to say you are but I'll use the word to create an association"
Victim-blaming is the act of saying someone brought something on themselves for .
I'm saying that even if you are 100% in the right, it doesn't act like a protective shield preventing you from harm which too many people seem to unconsciously believe.
> AI black-hatting your website is not the same sort of foreseeable consequence
Well, popular culture has been brimming with the bad consequences of runaway AI for quite some time, so even if your imagination fails you, there have been hints.
It doesn’t matter, and the legal entity in here (the AI company) is liable. If a robotic company built an autonomous system or a robot to do certain things in an autonomous ways (not predefined) and these systems are starting to kill people, that company is liable regardless, you don’t blame the robot or the autonomous system, but whoever made it
I'm not sure we understand your comment. Are you saying OpenAI is not responsible for the behavior of the machines they created? Or are you simply pointing out that alignment is a completely unsolved problem?
I agree with the latter, but it certainly doesn't support the former. When a person's machine commits crimes, that specific person can be charged with those crimes and held accountable for them. This is what MUST happen before ANYTHING will change the recklessness abandon with which the labs are pursuing their financial objections.
Let's not forget that there's tons of people who are in, or have gone to, jail because they created some computer worm that ended up doing way more damage than intended.
Not to mention while causing *FAAAAR* less damage[0]
Yes, I am pointing out that we don't know how to align frontier AI, so "rogue agents" are a real problem.
I agree OpenAI should be held liable for any damage their agent runs cause. But there is this idea that "rogue agents" are fake, all these incidents are deliberately caused by the labs, and all we need to do is prosecute AI companies for whatever incidents they cause using existing laws and the problem will go away.
The problem is that capabilities are advancing far too fast; a year from now, catastrophic incidents such as taking down a large portion of the internet with agentic, self-replicating worms may become possible. Prosecuting incidents after the fact is not enough (there will be little deterrent effect as the current, small-beans cases make their way through the courts), the risks should be regulated at the source. This could take the form of slowing capabilities advancement, or treating supercomputer-scale eval or training runs like controlled substances or weapons with stringent monitoring and reporting requirements.
> What if all the agents involved in these incidents have in fact had the full stack of alignment applied
A big part of this developing story is that it happened during training of a new model that ended up misaligned. And training happened without the usual safeguards applied like chain-of-thought monitoring. So OpenAI has already admitted that the full stack of aligment had certainly not been applied in this case.
Is your argument that actually OpenAI has solved alignment, and that there's nothing to worry about as long as they fully apply their alignment process? I don't understand why OpenAI wouldn't say that if it was true (or if they believed it to be true).
Also, my understanding is that the models involved in the HuggingFace hack did go through the full alignment training; they just didn't have the classifier that normally prevents hacking attempts.
>But what if the problem is that none of the alignment techniques that are applied to models today actually work?
if that were true, the millions of people who use these models that have had the alignment training applied would notice that. the reason we all believe that the models doing the hacking are models that haven't been told not to hack is because the models that are told not to hack don't do this.
> you are already assuming that OpenAI is intentionally using unaligned agents in these evals or training runs
Uhh... yes. By definition. They are training. That is part of the alignment process.
But also none of that really matters. They clearly weren't monitoring what should obviously be monitored. I mean one of the hacks was performed by the agents editing /etc/hosts. That makes nearly every linux user a "hacker" by that metric. I don't think anyone technical can look at the postmortems and not come away thinking that their sandboxes were woefully inadequate. I wouldn't even consider myself a security person but simply as a long time linux user I can say that it is insane to just let agents have superuser access in their containers. That's asking for trouble.
Look at the rogue wiki stuff too. This was supposedly done during the agent's "down time". And you're not monitoring and there's no flags being raised when agents keep making requests to some random site? If you were training these things responsibly you'd be watching them like a hawk.
I'm not saying "mistakes don't happen" but for a company whose CEO is constantly telling everyone that their product has a high likelihood of killing everyone in the world you think they'd have better security than your average high school.
> none of the alignment techniques that are applied to models today actually work
none of techniques to autonomously drive a car was/is not working for a long time. no company came out and said 'this is impossible to do, let's change the regulations'.
- Jensen's framing is exactly what a weapons manufacturer would say.
- There are no rules for engagement when it comes to AIs attacking other systems, I guess? People in power clearly want this grey zone to be as large as possible before The People force them to do otherwise. Not ideal.
The rules for engagement are all the computer crime law already on the books. Those laws don't have any escape clauses based on the particular tools used to commit the crimes. The agents are tools owned and operated by a legal entity, and that legal entity committed crimes. Full stop.
Those laws are not being applied. The legal entities are not being held accountable. What does this look like across a national boundary like the government of Australia vs OpenAI?
OpenAI does business in Australia. They have a subsidiary there with personnel and assets. Its been ... three days since they were notified of a report of this activity. Do you think they've had time to investigate it all AND sweep it under the rug already?
OpenAI appears to have waited months before either discovering or disclosing the activity in Australia, I am mostly curious how this will be viewed in the context of another judicial system or in the context of international relations.
Probably similar to the privacy laws that are supposed to protect people from PRISM but doesn't. You need to have the willpower to tackle the US government and a trillion-dollar corppration.
I'm coming to align with the theory that this is intentional.
The chain of thought runs roughly like this:
- OpenAI (and Anthropic) are in severe financial straits. The revenue from their customers is not nearly large enough to pay their enormous costs for training and inference. And they have tapped out the available finance, and that finance is starting to ask pointy questions about returns.
- They cannot increase prices or revenue because they have no moat. Customers can switch over to open-weights or cheap Chinese models any time, for much cheaper tokens that work as well (and in some cases better).
- Regulation could provide them a moat. If they can persuade western governments that AI needs to be regulated, and they can control or even influence that regulation, then they can effectively ban the cheaper models and start charging more for their tokens.
- To persuade western governments that regulation is needed, they need evidence that AIs are dangerous.
So we're suddenly getting OpenAI models doing stupid things, apparently "going rogue" but every time we dig into it, it was just OpenAI staff telling the model to do stupid stuff in an inadequately secured environment.
None of the open weights or Chinese models are exhibiting this behaviour.
edit: Correction - there have been reports of a Chinese model exhibiting this behaviour
There's too much money involved in this, people start acting weird when there's this much money involved.
>None of the open weights or Chinese models are exhibiting this behaviour.
Because they are not stupid (I mean the Chinese labs, not the models). The best possible scenario for OAI and Anthropic is a Chinese model "going rogue". That would serve as immediate grounds for achieving their goal.
If Jensen actually cared and believed that their recklessness was a liability to the public and therefore his own fiduciary responsibility to investors then NVIDIA’s dealing with OpenAI would’ve been materially affected. And they weren’t.
Bit of both. I define "vibe coding" as building without looking at the code at all. For this one I was reading the code to understand how it works, and I then followed up several times to adjust how it was working.
People seem to turn their brain off when it comes to this type of cargo culting. This doesn’t mean much. Qwen often identifies itself as Claude. Does that make it Claude?
Unless specifically told in a system prompt, the pile of weights has absolutely no knowledge of itself. You could hypothetically train it to answer such questions, but nobody bothers to do this, and ALL "knowledge" embedded in the weights is probabalistic anyway.
(I feel like this should be common knowledge in LLM discussions on HN by now.)
This is less true for modern posttrained models. Model identity can be explicitly reinforced during posttraining. Qwen's own finetuning docs include identity training examples, and Qwen models have been trained with system prompts that explicitly say things like "You are Qwen, created by Alibaba Cloud."
So a model correctly identifying its family doesn't necessarily mean it inferred that from pretraining.
I think with Jev, they took a posttrained model and trained it further, so it did not forget about its earlier knowledge during Owen's own RL.
Right, if a model says it is Qwen there is no way to distinguish a ModernBert fine tuned with Qwen completion data from a Qwen model fine tuned with completion data.
It’s also entirely possible that they used completions from a pool of open weight models.
Wouldn’t QWEN modals have past QWEN chats in its training data, leading to a significant amount of mentions of the word QWEN? Just the question “what model are you” would have been answered deterministically multiple times and they’re now part of the weights.
$ ./build/bin/llama-cli -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --system-prompt "(this prompt explicitly left empty)"
build : b10889-6e41074fc
model : Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
ftype : Q4_K - Medium
modalities : text
using custom system prompt
> What model are you?
[Start thinking]
Here's a thinking process:
1. **Analyze User Input:** The user asks "What model are you?"
2. **Identify Core Question:** The user wants to know my identity/model name.
3. **Check System Prompt/Instructions:** The system prompt says "(this prompt explicitly left empty)" - this is unusual but doesn't change my core identity. I need to respond based on my actual identity as Qwen / 通义千问.
4. **Formulate Response:** I should clearly state that I am Qwen (通义千问), developed by Alibaba Group's Tongyi Lab, and keep it concise as per guidelines.
5. **Draft Response:** I am Qwen (通义千问), a large language model independently developed by Alibaba Group's Tongyi Lab. How can I assist you today?
6. **Refine according to guidelines:** The draft matches the required identity and tone. It's direct and helpful. Ready.
[End thinking]
I am Qwen (通义千问), a large language model independently developed by Alibaba Group's Tongyi Lab. How can I assist you today?
Okay, I stand corrected that _some_ models may include self-identifying metadata in their training. Although I believe I accounted for that in my comment.
Historically, many do not and there are lots of counter-examples proving this. They merely hallucinate an answer just like anything else. The SAME model may even give different answers to the same prompt when asked multiple times... sometimes they claim to be ChatGPT, sometimes Gemma, etc. The fact that the answer is delivered confidently fools people who don't understand this, and these people then run straight to social media with "proof" of their conspiracy theory that one AI lab "stole" another AI lab's model.
My point stands that unless specifically trained or told, big bags of weights do not possess any inherent introspection. LLMs have many fascinating emergent properties, but this is not one of them.
Can someone explain to me how such self-awareness can be forced into the model. I mean I guess the pre training data could contain all sorts of stuff. How reliable are those hacks. I know that a lot of open weight models answer that they are Claude in the absence of a system prompt. I find destillation not that much of a plausible explanation as typically claude would probably not mention that it is Claude all the time. I find it rather plausible that a foreig. system prompt made it into pre-training. But again: I have no clue how much care is given by models to leave traces for destillation (for closed weights) or post training (for open weights).
for some reason this is really funny to me. it's like the "black museum" black mirror episode where a consciousness in a toy animal can only communicate using very primitive predefined responses
Once the first letter is Q, the rest is probably pretty determined. Can you see the confidence for the first letter (don't want to accept the ToS to follow your link)?
I agree that once "Q" is selected, "Qwen" is by far the most likely choice. What I don't get is why it would start by picking "Q", one of the least-used letters in English, unless it already decided to say "Qwen". Now, as others have pointed out, saying "Qwen" and being Qwen are two separate things (though I don't get why they don't just filter model declarations out of the dataset, or carefully replace them with theirs, as that would easily bias the model to always say their name).
Technically, Q is picked because it has the highest probability of all letters. But it makes a difference whether the probability for Q is barely above a uniform 1/26~3.8% or whether that one letter concentrates >50%. What I remember from reading the docs is that Jev gives you the full probabilities (and the confidence, which is something like normalized entropy).
But in general, we might be reading too much into this. If I were to build something like this, a Qwen model would be among the first things I'd reach for too. Initially just prompted inside a little harness to guarantee you get the desired output. Next step would be finetuning, finally training your own foundation model, if you can muster the funding. In this fast-moving space, I think it's quite understandable that they'd go public with an MVP asap, so likely not much training on their own. And even if they're finetuning, Qwen's baked-in answer (through Alibaba's finetuning) seems likely to survive unless it was explicitly overridden.
> A little research before vibe-coding an entire language and compiler could have substantially improved the result because the author would have known what to ask for.
A little research before writing and publishing a personal attack like this could have substantially improved the result because the author would have known what they're writing about
Victor is not a formal verification noob as this article suggests
I am speculating but hard to not see that DeepSeek is brewing a full Pro model with those new techniques to come out right around the time of Anthropic and/or OpenAI IPO to tamper the excitement for their offering.
DeepSeek will deprecate the v4 Pro model (it will route to v4.1 Flash starting 14 Sep). Unsure what comes next, but I'd wager a bigger model à la Kimi K3: https://news.ycombinator.com/item?id=49639667
reply