Games in this style have had a recent-ish revival with titles like A Tale of Two Kingdoms and Heroine's Quest. A Tale of Two Kingdoms is what King's Quest V would have been, had it been made in the modern-ish day.
Anything by Wadjet Eye Games is pretty choice as well.
There was a period where there just weren't any good adventure games getting made, but now there are nostalgic pixel-art style titles galore and many of them choose the IMO solid path of focusing on writing quality over realistic polish.
I have mixed feelings about the genre. I was there in the 80s at their peak. I loved the games but in hindsight I never really had a lot of fun because they were too difficult without hints. The only exception for me was the Quest for Glory series, which I beat legitimately back in the day.
A lot of the modern games are a hell of a lot more logical in their puzzles. I've played several of the Wadjet games, and the story lines have been always interesting, and I have been able to solve them without hints. They aren't like the old Sierra games where they wanted you to call a 1-800 hint hotline to make more money.
I think there is something missed in how you're thinking about this problem.
Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.
Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.
The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.
I don't present any support for or evidence of:
- LLMs gaining intelligence by whatever means
- Extreme intelligence being possible
- Extreme intelligence being feasible
But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
That's not what I said, and it's the least sophisticated way I saw to phrase it to engage your argument in a precise way. The closest thing to your summary that I would accept is, "words can't convince people to do arbitrarily crazy things, in a way that you can predict."
> If you had [superintelligence], you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X ... But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
You don't need perfect prediction, you need good-enough influence on enough of the population. Which is something that adtech, social and trad media, and troll farms have perfected already.
We literally have people running around convinced the earth is flat. And a president who is both senile and delusional with access to the nuclear codes.
Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.
Or possibly inserted itself into critical elements of the tech industry. A digital adversary could do huge damage just by taking AWS offline.
That's just one possible attack vector among many others, most of which aren't well-defended. Especially if the AI itself is being used to harden their defences.
By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?
The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.
> Especially if the AI itself is being used to harden their defences.
Your use of "is" rather than "isn't" only makes sense to me if the former interpretation is intended. To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented. (Unless you mean the bad actors will be the ones in control of all AI, i.e. OpenAI and Anthropic and friends?)
> A digital adversary could do huge damage just by taking AWS offline. That's just one possible attack vector among many others, most of which aren't well-defended.
I think you picked AWS because it's one of the few high-consequence digital systems a rogue ASI could plausibly take down on its own. Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.
Note: This is a very good argument to never remove the air, time, and meat gaps from those systems. If they do get removed, it will not be the genius of AI, but the stupidity of mankind that dooms us. Even Qwen could kill us at that point :)
>By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?
The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.
You seem to be moving the goalposts a little, but I don't see why "rogue AI" is less credible after the LLMs. It's obviously more. And we're just at the start of constantly running agent style harnesses.
>To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented
And yet, in many cases, those on the offense have a big advantage that any defense lacks. Especially if there's no concern for one's own life, which whether it's an AI or a human AI user removed from the field via his use of AI to delegate doesn't need to have (or they're just mad/fanatics and don't have in general).
>Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.
> Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.
Or like a massive AI datacenter boom of questionable long-term profitability? Maybe the end goal isn’t profit but large amounts of compute?
>So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
And why do you need perfect prediction and/or perfect control? "Good enough for the results you want" prediction and control sounds perfectly fine.
We have had cult leaders that had neither perfect prediction or control, and still convicted hundreds of thoysands of people to live as they say, and a decent chunk to even suicide on command!
> (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|
"You can't know what the impact of your action will be if it doesn't change much"
> (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large
"You can't necessarily change much"
> (or that f_actual(X) saturates its codomain)
"People can't do arbitrary things". I agree with this statement, but... people can build atomic bombs by themselves? See The Atomic Boy Scout. I don't think it's very relevant to the discussion that you can't make someone phase through a wall.
> when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
"Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".
What people think of as a very strong password often has <100 "security bits" of entropy.
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
That makes it impossible to steal the key if you MITM, but there is still the same chance of guessing it. (Assuming password has as many bits of pure randomness, such as machine generated password.)
The best cryptographic key in the world would be defeated by a breach of your filesystem.
The comparison here is between in-memory cryptographic keys and in-memory passwords. Yeah typing a password into a keyboard adds a vulnerability in between the keyboard and the computer, but it's not a huge difference in security posture. If the device using the secret is compromised, so is the secret, whether it came from the user's mind or not, whether it was stored encrypted before the user used it or not.
A physically separate hardware security module does add protection but I think that's a different conversation.
Worth noting that if you are running a remotely recent version of SSH it has everything you need built-in. No need for Fail2Ban on SSH these days if your config is fully modernised.
`PerSourcePenaltyExemptList` is also worth a mention if its the sort of thing you like, we don't use it at $work as whitelisting IP ranges is not really in-line with modern security best-practice (zero trust etc.).
There is regular bot traffic attacking SSH, but it's not "saturate your downlink" level of traffic. If you're getting attacked that heavily then you're being specifically targeted.
Huh? I run ssh (OpenSSH) on port 22 with root login enabled and no fail2ban on a single-core laptop cpu from fifteen years ago. The load average is 0.00 even with the small mail and web server that runs on it
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)
Working as a sysadmin, and with disk encryption the constant log writes generated from ssh was enough to cause heavy load on severs. Long lasting connections from bots also forced us to increase the maximum number of sockets, as some scan waves was enough to hit the maximum and block other connections. Fail2ban has mitigated those issues to the point where neither has been an issue.
I believe this depends on your exposure level. If you have one, or maybe no, domain names pointed at your server, you are probably fine. If you have tens, or hundreds, of domains pointed at your server (i.e. you are a web site hoster), you will probably need something like fail2ban.
On another server with the same setup, there's various services and websites running, so load average is never zero there (due to scrapers) but ssh has yet to catch my eye in htop. Whenever I've looked, even the amount of cpu that a Windows VM uses constantly while idle is many times higher than what ssh uses
Another way to view this is as metonymy. "Google" is a metonym for "Google Staff", so using "their" to refer to the humans-making-up-the-corporation (as referenced by the metonym) is appropriate.
I agree it's correct English to say "Google have launched their new product".
If you want to be even safer, you could use a physical network-connected KVM. The only interfaces those expose to the connected computer are HDMI (graphics read only) and USB peripherals (write only). Sometimes they can also physically power off the hosts to which they're connected.
It's a pretty darn slim surface for an attack. Technically doesn't meet the definition of an "air gap", but it gives you 100% of the utility of remote work with effectively 0% of the vulnerability, at the cost of a tiny amount of power and a few extra cables in the rack.
The simplest way to "prove" this is to show that a model can emit an image that isn't part of its training set. I'm sure you've seen that a million times already - what is a "fake" if all "fakes" are regurgitation of a real image?
That means models are capable of outputs that were never their inputs.
It's easy to see why this happens if you think of training as selecting a target against which the model will be scored, and then randomly mutating the model until it is close-enough to the target. There is no guarantee the random mutations produce something identical to the target for ANY of the training data. A model might finish training at at average delta of 10%, and it's possible that means that model is incapable of exactly producing anything in its training data (it could be 90% accurate for any prompt, or it could be 100% accurate for some and 80% for others, or...).
In addition, prompts that were never present in the training data might produce horrific or random outputs, because those were never even scored during training so they might get something right-ish or something entirely full of garbage data. No telling.
And this is all ON TOP OF the nondeterminism in inference. Everything I said above would still be totally true even if inference were determinstically done.
The question is "why are the agents emitting tool calls that cause communication". It's not some kind of "desire" fueling it, it's probability conditioning. And in this case it came primarily from the context window with words that are weighted toward communication-tool-call output, not some kind of bias in the model's training.
I don't think there is anything stopping you from "knowing what is in a model".
For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).
The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.
I see no reason the same principles couldn't apply to understanding any other model's behavior:
- "open weights" (you have access to the trained parameters)
- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)
- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)
I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!
> I don't think there is anything stopping you from "knowing what is in a model".
I don't think knowing what is in a model is particularly useful.
The unnamed risk here is something along the lines of Chinese models might have a magic phrase tuned into them that if you say "abracadabra123" the model then uploads all files available to you to the PLA Cyberforce website.
Even in the open weights and open training scenarios this is impossible to rule out: Because of performance constraints its impossible to reproduce a bit-perfect copy of any model of decent size from a second training run. Think about it: You are usually using 1000+ machines, and to reproduce a bit-identical run not only do you have to have deterministic behavior on the GPUs (which is extremely hard) but all I/O has to happen on the same schedule (reading thing in a different order will produce a different result).
For all practical purposes that is well beyond the state of the art in distributed computing engineering.
This is born out by reports like the Olmo training:
> DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues.[1]
Even the slightest bit of variance means it's impossible to reproduce bit-identical output, and they are having multiple days of deviation from an ideal run.
The goal here isn't "knowing what is in it". Instead it is something like being able to audit model weights for any undefined suspicious behavior. That isn't something that is possible.
Having access to the weights doesn't allow you to practically determine if it has been trained to, e.g. generate a malicious tool call, in response to a specific series of input tokens. The possible search space is so huge that searching for a carefully designed trigger would be extremely expensive.
The GP's claim was not that open-weights models aren't understood. It was that NO model was understood.
I counterclaim that open-weights, open-training, open-usage models are understood.
You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).
> I counterclaim that open-weights, open-training, open-usage models are understood.
Knowing what is in the model is a long way from understanding it.
The entire field of mechanistic interpretability exists to try to understand models. Read for example "Open Problems in Mechanistic Interpretability"[1]. To quote:
> Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system.[1]
(emphasis mine)
Here's a practical example relevant to this case: Is it possible to monitor a LLM (the LLM activations themselves, not the actions) for cases where the actions it is about to perform (eg upload files to a PLA website) are not reflected in the goals of the prompt and are not visible in the thinking trace?
At the moment that isn't possible. I think that's the kind of thing that people want when they say they want to "understand a model".
> Knowing what is in the model is a long way from understanding it.
The claim I was arguing against reads as follows:
> You don't know what is in any model, open source, closed source, from China or from the US.
I contend that you do, in fact, know what's in a fully open model. I don't really get why the goalposts need to move beyond that to something I wasn't saying about bounding the potential outputs given unbounded inputs and nondeterministic choice of inference parameters.
I'm literally saying that if you see what goes into the sausage it's foolishness to argue you can't know what's in the sausage.
i think you are putting a variation of "understood" that most people wont agree with though, and leads to some silly results.
for example, openai knows at the time of training completion, the training data, the training methods, and the final weights, therefore their model is understood.
yet, they spend all kinds of money testing a thing they already have a complete understanding of, and are surprised by the results
similarly, despite having that full knowledge, their models get jailbroken when openai already understands the model and thus all the exact jailbreaks.
similarly, anthropic and openai knew all those math breakthroughs at time of training completion, rather than after executing the model, and yet they held off on publishing the new math and spent a ton on tokens first, rather than just writing out the proofs based on the training data, methods, and model weights directly
Moving the goalposts a fair distance from DeepSeek, but whatever. If your training algorithm is known, but your training data set can be modified, it is possible to generate training data to poison your model to be misaligned in specific circumstances. Poisoned data is not obvious. Good luck determining if misalignment has occurred.
Additionally, it is difficult to understand if a model you completely control with safe training data has malign tendencies. It's an open research question why this happens. There is a good explanation of this by Owain Evans:
I think they are referring to that LineageOS by and large (there are probably exceptions) does not have support for relocking the bootloader. Some apps refuse to work with an unlocked bootloader (but there are ways around it).
Anything by Wadjet Eye Games is pretty choice as well.
There was a period where there just weren't any good adventure games getting made, but now there are nostalgic pixel-art style titles galore and many of them choose the IMO solid path of focusing on writing quality over realistic polish.
reply