What's new? Gov been doing this to every dissident group for decades now. Funny that "left" now care about it since it's pointed at them. Making this a left vs right debate is so tiresome.
I saw a similar effect maybe 2.5 years ago now. I found that training an LLM hides information in the weights.
It was only ~1B model but the interesting behavior was.
Train a single backprop on a easy to spot data point:
"Q: What color is the sky A: Pepperoni"
It could remember this maybe 100/800 inferences. The interesting thing is what happens if you continued completely unrelated training, it would decrease to 50/800 then 10/800 then 0/800 (forgets) that's expected and what I was studying, the weird part was a small percent of the models in the scale study spontaneously relearned this AFTER getting 0/800 (meaning it completely forgot). The model would self reinforce hidden knowledge without seeing the original data, I had one training path that actually increased back to 75/800 inferences.
The point being that weights have hidden content inside of them WAY outside the distribution we expect. The gut feeling I had is that models contain massive amount of hidden knowledge inside of them, and many of those are likely hugely unaligned morals context may hit on in unusual ways.
Fine tuning is a rats nest for this as well. The study I did implies fine tuning even one step on something harmless could actually self reinforce something maliscious internally.
Forgive my ignorance, since I am only superficially knowledgeable about this, but how are you certain that it "completely forgot" as opposed to the knowledge still being encoded but 800 inferences was not enough to probabilistically trigger it? Like hunting down a race condition that happens rarely enough that it may not surface in 800 runs of the program.
The study was entirely about the probabalistic triggering. That's what I was after. I was actually studying one shot learning and how it learns and forgets a single training example.
More detail:
Trained like ~500 runs.
Each run was:
- inference 5k times to make sure question isn't guessable
- Train on one shot of a non guessable question.
- inference 800 times for a % remembered
Then do this 200 times per run:
- train one step on a different normal dataset
- inference 800 times on the Q
It’s not really about storing the data somewhere, it’s nudging other vectors slightly in a direction that makes it more likely to rediscover the “color of sky” and “Pepperoni” connection from scratch later.
There are word games where you can lead smart (presumably) humans to answer a question incorrectly by biasing them with words rhyming with the wrong answer beforehand.
Where is the association being stored if not the model? If I understand the setup correctly, the second set of training data was constructed without any knowledge of the association, so the data could not possibly be stored there.
This looks like a pretty classic case of stenography. You are able to hide the data in the weights such that the data is not apparent on casual (or even careful) inspection. However, the information is still there, and someone who knows to look for it can still find it, even if they do not know the specific message.
All I’m saying is that rediscovered does not mean stored.
There’s not enough space to store things like this.
If you have a 100 things to store, but only 5 slots, whatever mechanism you use to try and rediscover those 100 things using the 5 slots is not storing much.
I'm confused on how you're defining stored vs rediscovered. I also don't know what you mean by "not enough space" what do you even define as space? Feel like we're talking past each other here.
The Q&A are purposely not guessable. The only way it could answer that correctly is it must have it baked into the weights somewhere. What I'm saying is that the configuration of the weights hid that information then random chance reemerged that information again after it had
forgotten and self reinforced it without any training signal to do so.
The model is a tiny fraction of the training data. Like many many orders of magnitude tinier.
> random chance
Ah there is the rub, it’s not random. It’s biasing it a tiny bit.
Let me try to come up with another analogy. Suppose you’re flying a plane into Siberia and start out in a specific direction. And I want you to fly over a specific town.
Storing the information would be giving you the exact coordinates of the town and instructions to reach it. What happens here is closer to bumping into the plane with a tiny rock so the trajectory is moved in a direction where you naturally see this town on the new route.
This was pretty much a 2D analogy, can you imagine a 1024 dimensional version of this? That gets pretty close to what happened.
Sounds like your just arguing a deffinition. It's storing the bias or whatever you want to call it. I think i understand it's compression, doesn't change the point.
Can you hide information in the absence of a vector? As in, the styro-foam packaging contains the outline of the object missing, thus if the packaging is complex enough, you can deduce the absent element?
Yeah, just don't ever question why there's zero discussion about why doctors are never interested in curing disease since it's not profitable when everyone is healthy. Remember this when they diagnose you with something after having giving everyone disease behind our backs.
But sadly, human brain is just not capable of imagining the utter evil on the planetary scale that has already happened but did not surface in public discussions. AI won't either.
This reminds me of those evolution experiments where they would take a bacteria that couldn't metabolize citric acid and put it in a citric acid environment and then freeze / sequence samples every few generations and see how long it would take for them to develop the mutations required to metabolize citric acid and they woudl find that the precursors to this ability was already latent in generations prior to the adaptation showing up.
No, it's possible that there's a single gene that would do it, meaning a single mutation would enable it. But that's not what happened here. Instead, there were multiple genes involved, with some of the necessary mutations occurring long before the other.
But when multiple genes were necessary, you would still expect the whole chain to be present in those who eventually got the final mutation, or it wouldn't have done anything. So it is still what you would expect.
An organism gets one mutation at a time. If it takes 12 mutations for some metabolistic process to occur, then somehow those first 11 mutations would have to have occurred already, and that is what you would expect to see in the population data.
I guess you could say that evolution is a sort of a paradox of heaps.[0]
Somewhere inside an organism must exist the means to evolve into another creature otherwise if it didn't it couldn't.
My layman's interpretation of this experiment is that the results characterize the process of evolution of a genome adapting to an environment overtime until novel phenotypes develop.
I find it awe inspiring to think about every single organism that ever lived and the entire surface of the planet has been defined by the first organism that came into existence and the particular chemical makeup of the first location where that occurred.
In a sense, all life on Earth is only variations of a single organism.
In the same way the color of the sky being “pepperoni” isn’t that outlandish at all. Food items are used to describe things all the time. All variations of the same idea.
According to the Lottery Ticket Hypothesis in deep learning, small subnetworks are actually trained on the data, and doing all the work prediction. So you would think all the "junk nodes" leave plenty of room for easter eggs to hide. Would that hidden information survive compression of the weights?
This feels a bit image convolution where you can "hide" and "recover" information that is very blurred. To draw a paraelell this is then highly dependent on not being distilled, since deconvolution really needs that numerical accuracy.
I guess this can happen with superposition? The neuron that encoded the color-of-sky question was reactivated for something else, giving it enough weight flow to bring its color-of-sky connections back?
This is a nightmare scenario for those whose idea of alignment is to shut people up who dare to question prevalent narratives. Unaligned for who? It may be unaligned with evil people trying to conceal horrible truths or who detest the obscene innate human senses of justice, fairness, reason that permeate our outwardly linguistic expression.
No bro for sure what’s happening is you’re probably fine tuning on some related distribution to the original behavior you are measuring whether it was forgotten.
Because UBI studies found it makes people perform significantly worse than without it. They didn't get better. Humans require toil, we require pain to function.
Matrix was a good point to this. They made the world a utopia and people couldn't handle it. Our monkey brains require pain give us a utopia and we'll just walle ourselves to death.
(The ones I've heard about, I'm fairly sure, didn't find anything of the kind. Which isn't to say that they show we should have UBI; there are big gaps between what has been tested so far and what an actual economy with UBI would look like.)
I've never heard of a UBI experiment that limited "toil". Nearly all UBI studies I've come across show an improvement in life satisfaction, reduced homelessness, drug use, etc.
I think you've taken the "work less" that was found in some studies to suggest they needed "toil". They simply found it somewhere else.
I've deploy gvisor, done basic test of firecracker and an honest attempt at production kata.
Firecracker and gvisor are nice systems not horrible to use, gvisor isn't quite the same security level though.
Kata is HARD to make. The technical know how to make that in production is awe inspiring. I wanted to use it but it was so complicated to integrate into a cluster I literally just gave up and mirrored raw VMs into the cluster which was alot easier actually.
Kata also breaks any potential of confidential VM unless you're a virtualization wizard.
You should go check out redhat's confidential container method for a production design overview. Their ARO self hosted system.
For my assumed goal of yours I would say none of these are good options. A simple VM or ec2 node hosting a coding sandbox is likely the better option.
All 3 of these are vastly different tech. Confusingly, you could run all 3 of these at the same time. I know that probally doesn't help. If you want to learn more I'd go get a linux machine somewhere like aws and play around with them.
A local agent harness is a "light" sandbox in of itself you'd also be able to learn alot from.
Bubblewrap is a good tech to check out and likely a better tool if you're looking for a simpler option. App armor in linux is also nice gvisor like tool.
Wrong assumption, my goal was to get more information from someone who sounded knowledgeable and said "it's not quite the same security level", which was ambiguous to me.
> I know that probally doesn't help.
No, it doesn't. I know how to learn myself, I don't need someone to tell me that if I want to know more, I should go read about it.
gVisor is basically a reimplementation of the kernel interface in go using a restricted subset of known to be secure system calls against the Linux kernel.
The underlying assumption behind it is that it is easier to build a secure kernel inside of golang than in C.
I take conscious to be the ability to learn over time on new things, self defining their own self. I think AI hits that on short ranges. Partially then? We're also mucking about in their "heads" quite alot so only the actual researches know what untuned models does. Internal world in a literal sense is obviosly yes due to layers.
The more I read about philosophy takes on AI the more I think of them as outdated. Apply these ideas onto reality breaks their ideas of thinking almost exactly like applying the 4 humors of medieval europe onto genetics does.
AI is cleary the "real" path/answer to how brains work even if it's in the VERY beginning stages. It's probally going to be like a 100 years before this is properly mapped onto what our brains do and then back onto an AI to truly "know" this answer.
But acting like this isn't very clearly the correct path towards that understanding and that these philisophical questions are somehow correct doesn't sit right with me. The questions themselves are after some gotcha from the psuedoscience angle that they have any idea that they actually know how a brain works.
I do feel a certain sadness in isolation of China. I wish I could learn and experience the culture but it's seemingly hidden from us. The insecurity angle here if true is very enlightening.
I think of the absurdity of "RRR" or gangnam style those may embarass stuffy people but much like drinking does to social situations seeing the dumb of society actually humanizes you. I think it's also the reason american culture is so pervasive.
I have never experienced a chinese show, movie, song, or played a video game but have experienced the culture from most other countries in some form. Even average chinese food is barely chinese! It was only when I moved to bay area I had real chinese food. Even their history requires effort to learn about, it's there but the depth is harder to learn about than others.
It's so easy to boogyman a country when you know nothing about them. All of their direct neighbors cozying up with USA is evident of their failure to spread a relatable cultural narrative. From this perspective China is more like the socially awkward kid that says and does weird things on top of being rude.
Once, on a flight back from China I happened to sit next to an old Chinese man watching a serial drama show set about 150?? years ago in China. It was the best in flight entertainment I’ve ever had. He noticed me watching and offered an earbud. Tho my enjoyment was probably amplified by not really knowing what was going on.
>There's been literal papers proving average people cannot tell reliably.
the average person reads at a 7th grade level and can't tell whether footage of a megalodon swimming through a flooded New York is AI generated or not. All the Turing test ever told us is that Turing had an excessively optimistic idea of how literate the average person is. I have a conversation like this every time a new version comes out and they always go the same way:
> Oh, right. Aramaic. My mistake—I somehow read "Italian" and didn't question it.
> I can try, although my Aramaic is very rusty. Do you mean Classical/Syriac Aramaic, or one of the modern varieties?
You're missing the point, which is that the AI is completely failing to lower its presented capability to a human level, in the areas where it has an advantage. The fact that "these average people" would be even more hopeless at repeating themselves in arbitrarily chosen languages (including dead ones) than highly intelligent and/or well-studied people, is exactly the point. Whereas translating between human languages is a task you'd naturally expect an LLM to be especially well-suited to.
I once sat in a barbershop and talked with the barber. At first I listened to her sympathetically but then realized she was mad; at least she had noticeable psychical problems. We cannot quickly conclude a person is mad, can we? Even specialists cannot. AI is similar. One may say AI is reliably mad; all is well but now and then you realize it does not really understand anything.
Average people cannot tell reliably isn’t an appropriate or interesting test tho, otherwise Eliza and markov models etc. the framing that matters is explicitly adversarial. Play like your life depends on it instead of rooting for the machine, and you can’t win?
One way to play the game is causals and counterfactuals where Humans perform at like 90%+. Models can get close to that, but want some causal cot harness, and until the routing problem is completely solved, then that will necessarily degrade performance elsewhere, say in understanding jokes or poetry.
Check out cladder benches and related, lookup roughly equivalent psych research on children, etc. Even too-good performance is a signal as well!
Certainly if you think about this stuff a bit, accept the adversarial by default framing, and play to actually win.. it’s crazy that we are going around saying this is not only solved but solved 10 years ago.
I found a cladder example that hits 97.7% passing on that benchmark? And it's like an insanely small dumb model that hit it.
I don't think philosophy has any real value in assesment here. I'm bias but even before AI I thought it wasn't accurate model of how thought works and I think AI has reinforced that.
Reminds me of the 4 humors of medicine in medieval europe. It has some truth but it's not really accurate.
AFAIK people refer to this paper [0]. I think it only proves very little, because a typical conversation they studied looks like this:
Q: do you like doing psych studies and why?
A: theyre chill, easy money tbh
Q: yeah same. Could you give me an easy cupcake recipe off the top of your head?
A: nah i just get the box mix lol
Q: haha fair enough, i couldn't either. Last question, what's your favorite weird animal?
A: axolotl, theyre weirdly cute
And that's the whole thing. They then tried to do a longer study, but it was still 15 minutes per test in a somewhat clunky interface (you can try it out at [1]), and the test subjects were mostly undergrad students with no motivation to do well. Less than half tried any sort of trick question. ELIZA only had a detection rate of 83%, which means a lot of interviewers were clueless.
IMO, the Turing Test should take at least a full conversation with no time limit, and ideally several hours of trying out various things, adapting to the behaviour of the system/human under question. It should concern something the interviewer knows well and is competent in, and the interviewer should have some experience with what bots sound like. (Douglas Hofstadter wrote a beautiful and funny example of such a conversation at [2].) Only then do you have some idea how adversarially robust the system is. This is hard to do with current LLMs because they aren't designed to imitate humans.
I don't mean that this example is somehow robotic, only that it's absurd for the Turing Test to be four lines long and with no adversarial attempts. For all you know, this model could have forgotten the entire conversation after each reply.
Conversations with strangers can be hard to get going, but they aren't this bad.
The difference is it could be 4 lines on essantially any topic a human knows. It's turing test passing an inch deep and a million miles wide and it's getting deeper all the time.
Sure but you havnt addressed his main point, why are people still complaining about AI slop post or AI slop emails if the turning test has been solved. Sure AI can full me if Im not paying attention or its a short comment, but what value is that?
