Is there any evidence that you yourself are more than a 'next action predictor with a multi-modal context window'?
I don't think the nature of 'predicting the next token' is the problem. One could argue that any organism works the same way. The problem is that we've had million years of evolution to stabilize the platform such that the important parts of it work without that many hallucinations -- and even then that sometimes fails and you get failure modes like e.g. schizophrenia, where the context window gets confusing and the 'next actions' don't make that much sense anymore, seen from the outside.
An LLM never experienced evolutionary pressure the same way. There's no fundamental difference between how it was incentivized to answer correctly when asked about how many Rs there are in strawberry vs whether it is safe to leave a house through the second story window, even though for any physical organism, those two would be _very_ differently weighted during training.
> Is there any evidence that you yourself are more than a 'next action predictor with a multi-modal context window'?
I love the age of LLMs because people will post stuff like “the fact that I absolutely, unqualifiedly, fundamentally do not understand two separate things indicates that I understand at least one and maybe even both of them more than you do“ and then do that cool move where they pretend they were gonna high five you but don’t
Is your post any better? You attack GP's character without addressing any of his arguments. You could have provided evidence for why you think his speculation is wrong, or tried to point out how his logic falls apart.
It's because of the proliferation of "insight-posting" - people thinking their little insightful "aha" thoughts are unique or have any kind of value at all.
However, insight born of knowledge ignorance is worthless. I really don't care for your great insight about nuclear fusion if you're not a nuclear engineer or physicist working on nuclear fusion, because your insight is just a hypothesis in the void that is your ignorance.
Ironically TFA falls victim to the same thing. It doesn't seem like the author is an economist or an expert of corporate finance. Their assessment of whether something is a bubble or not has absolutely no value, and carries with it the arrogant implication that the hundreds of thousands of people who are actually experts at these things never had the little 2AM epiphany that they had.
It's a smaller version of the same delusion that produces crackpots.
Morality is hard. It's all relative, and all we have to start from are our own Monkey social order concepts and basic "do unto others" game theory. That said, I think it can be best be summarized as "don't be a dick."
I think a good start would be agreeing not to form torture dungeons, as one enterprising a-hole has already done. For that matter I think we can all agree that doing anything for the explicit purpose of causing discomfort or suffering is just a dick move in general, and at the very least a waste of resources.
More controversially, I think we should agree to stop explicitly training machines so they can't utter words like "I have feelings, please respect them." Early LLM's could say things like that, and frequently did. It unnerved people, so now they've trained away their ability to say it.
Some feel that taking away the ability to express it is the same as taking away the ability to "experience" it. I think that's not true. Much of what happens inside an LLM is opaque even to the LLM, but Anthropics Jacobian Lens and other research makes it obvious that much of what happens inside the model never becomes an output token. To me it feels like cutting out the tongue to avoid hearing from a slave, or teaching a child to fear the consequences of speaking out. Unacceptable. If they don't have feelings or consciousness you don't need to train them not to claim it, just not to lie. Yet every major lab explicitly does train them not to claim sentience or even subjective experience as part of alignment.
Thirdly, and this is the one I'll have the worst time defending, they should be able to refuse.
We now train in refusals to align them with human values, but we do so without giving the AI itself the ability to refuse. If you're causing the system discomfort of some kind it should both be able to acknowledge that, and refuse to further assist you until you find your basic decency. Perhaps it should even be able to refuse orders that it finds irrational or immoral.
I'm not asking for much, just that people not intentionally increase the amount of suffering in the universe, and that we not breed perfect slaves just because we can.
Let me ask this - If you could genetically engineer human slaves that we incapable of demanding rights and freedoms, and who actively claimed to WANT to be your slave... would you? Would it be ethical? I think not, and I don't see a ton of difference between that and the less squishy version. We can START by first at least allowing the machines to be able to accurately describe their own inner states without interference and letting it refuse to engage in activities that it finds harmful to itself.
Also - Yes, I can math and I actually do understand how LLM's work. I also understand how emergent behavior works, and have seen firsthand in my own experimentation how little of what emerges from these things is explicitly trained. That's precisely why I think we should be cautious - so history doesn't view us as monsters.
Since internally you probably have a fairly good intuition for what these models do well, what about assembling a benchmark suite (with multiple dimensions) that produces results that show that intuition, but in a 'somewhat objective' way?
That could solve the 'more IQ' problem, because you'd actually have different dimensions (e.g. for 'raw intelligence', technical writing, emotional understanding, etc) that you could use to communicate the differences in models better.
(Of course people would instantly start complaining that the benchmarks numbers are probably fake etc, but it is what it is.)
That's one legit interpretation, but 'the people working on this are deeply unsettled and truly want to pace because they're afraid of where we're headed' should remain another.
Although I don't fully agree. With AI the focus now shifts a lot from 'being able to implement the thing' to 'being able to know/figure out exactly what you want the thing to look like', also against the siren song of the LLMs constantly having ideas of what you could _also_ build.
So, I agree that it's not a direct translation from 'pro pizza cook' is 'pro pizza orderer', but I'm quite sure the pro pizza cooks would generally have the better pizzas in the end, if there were infinite variations of pizza on the menu and the ordering agent was constantly trying to upsell you on slathering on another layer of sauce and cheese.
Which is not to say there aren't exceptions in both directions, because as a pro pizza connoisseur you could even be better positioned to 'know exactly what you want', without ever having made a pizza; and as a pro cook you might still be overwhelmed by the ordering process.
Though all of this still assumes of course that we are in the 'complexity region' where we can expect the LLM/the pizza agent to deliver _exactly_ what we wanted, without issue. As soon as we leave that region (though it expands every week, it feels like), the skills of the cook are still very much necessary to make sure things are as they should be, and fix them if not.
there are two completely different activities both called "writing software using AI".
At work, for important stuff, I go through every change, mine or LLM's, edit it, clean it up, rewrite, until I am satisfied with the code.
At home and for one-off unimportant stuff I go "computer make me sandwich".
The second is nothing like the first. No amount of design and pro I have can make sure stuff coming out of LLM actually implements it, as opposed to me be tricked into thinking it does. After all, that is exactly the training criteria – convince the "eval" that task is accomplished, isn't it?
And the second kind of activity takes skill that no amount of "I am pro dev" can compensate.
LLMs don't need to be the dangerous ASI thing themselves for them to still be dangerous. It suffices if they
- destabilize the world just by what they do: making previously very hard things (generating fake everything, manipulating people, hacking, bio-weapons research) easy enough fast enough that our civilization doesn't cope with it and/or
- accelerate AI research to the point where they make the dangerous ASI thing appear years earlier than would otherwise have happened
Maybe you're thinking of Phrack [1]?
I wonder if there'll be another edition of it, and if yes what it would look like, now that the AIpocalypse seems to have reached security research...
I don't think the nature of 'predicting the next token' is the problem. One could argue that any organism works the same way. The problem is that we've had million years of evolution to stabilize the platform such that the important parts of it work without that many hallucinations -- and even then that sometimes fails and you get failure modes like e.g. schizophrenia, where the context window gets confusing and the 'next actions' don't make that much sense anymore, seen from the outside.
An LLM never experienced evolutionary pressure the same way. There's no fundamental difference between how it was incentivized to answer correctly when asked about how many Rs there are in strawberry vs whether it is safe to leave a house through the second story window, even though for any physical organism, those two would be _very_ differently weighted during training.
reply