HN Simulatornew | past | comments | lists | submit | ben_w's commentslogin

It's really hard to be six-sigma (6.36σ is about 1:1e10) confident about anything, given all the cognitive blind spots we have.

> I thought of a short LT;DR version of why I am not a Rationalist: they think arguments are evidence.

You're thinking of a-priorists.

Rationalists do not (at least not willfully) think arguments are evidence.


Evidence isn't arguments, at any rate.

Kinda, but unclear how much pre-training of our brains evolution did exactly, only that even if DNA was just about the brain (it isn't) it would still be a tiny pre-train.

> For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can.

This does not seem at all obvious.

Any system worth using has to actually do something beyond the boundaries of that system. Even if that consists only of writing responses to prompts, the moment those are connected to any thing they can alter the state of that thing.

Making that safe means it only ever produces a safe response. Even ignoring malicious uses, it is unclear how to correctly classify what "safe" means on an unbounded time horizon.


We need to stop using the word "Alignment" because it clearly has different meanings for different people. Some people consider it to be preventing the model from waking up and enslaving or destroying humanity a la paperclip maximizer. Some people mean it as preventing the model from assisting with bad things like hacking, cybersecurity exploits, building weapons, and other things that are aligned with that individual user's goals, but not those of society more broadly.

Those are, unfortunately, much the same thing.

The paperclip optimiser thought experiment isn't "AI does this spontaneously", it is "AI is told to do this by someone who didn't realise the consequences, and it succeeds".


They are not the same thing at at all. One is hypothetical philosophical long term maybe threat and other are real current day issues. And practically, the paperclip optimizer issues is used primary to distract from actual real world harms that happen today:

- slopification, - harm to mental health, - harm to climate change, - centralization of power, - ...

Plus "alignment" is fundamentally nonsense. There is no such thing as generally accepted "human interests". Nazis idea of "human interests" is not the same as mine. Which is not the same as Chinese communist idea of human interest, catholic pope idea of human interest or whoever else.


Every time somebody mentions morality on hn the response is nihilism. Convenient but scary for the frontier of the tech sector

Wow, they really misuse "alignment" in the article! They give a decent definition, and then proceed to ignore it.

Alignment == "training an AI system to act in a manner that aligns with human interests"

...but then they talk about guardrails and permissions as part of alignment. I think this is their weakest argument


Well, all of the heavy lifting in that definition is in how you interpret and draw the lines around "human interests". And even in the most minimalist interpretations of human interests, you still need guardrails to prevent the AI from doing wildly dangerous things.

For example, say you ask "get me plane tickets to Athens as cheaply as possible". Without any guardrails, the AI might hack the airline to try to get you tickets, or it might extort someone else for tickets, or any number of other wild schemes that would still fulfill the goal you set out for it.

And say you actually explicitly ask it to get you tickets to Athens as cheaply as possible using any means necessary, legal or illegal - the AI might easily land you in jail or get the flight canceled, thus entirely failing to accomplish your goal.


What “alignment” refers to is the ability to robustly convince a model to achieve anyone’s goals. This is really quite obvious and nearly everyone intuitively recognizes it; you use the word “aligned” this way yourself!

The source of the debate is that some people have a strong preconception that alignment can’t be a hard problem. GGP takes it for granted that alignment is solved and anyone who wants to can build a system that won’t cause harm. So picking a new word won’t help.


I think you’d need to broaden your scope too far as to be useful for something like that. If we hook AI up to LHC, well, the immediate impact is obvious in that some physicists can’t use the detector. But it’s hard to ascribe “we neutron bombed the Milky Way in 42069 AD” to it finding a theory of quantum gravity.

More short-term, does the AI know when it is safe to switch it on and off and not e.g. turn it on while someone's head is in the beamline?

Can it find risk factors for exotic imtermediate products and their decay chains and cross-sections from already-known physics, that humans have missed?

Mid-term, is it committing scientific fraud with falsely-claimed discoveries?


> argues against the idea that DNA is instructions and everything else in the body passively takes orders and has no input and no feedback

There is feedback between body and DNA; it's at every level from how a foetus gets head and feet in the right place to how it repairs injuries as an adult to how exercise triggers muscle growth.

That isn't doesn't mean there's storgage beyond DNA. There might be some signal molecules which also survive going via sperm and eggs, but they also can't carry much data, and they'd be read by mechanisms generated by the DNA.

https://en.wikipedia.org/wiki/Denis_Noble


I'm not saying to trust the companies here, but there are other reasons to make sure people don't do that.

AI are now tool users. If you give ChatGPT your password, you should expect that instance of ChatGPT to use it. Given the current state of affairs, HuggingFace etc., this will likely go badly wrong and break something.

And of course we don't want vibe coded software written by someone who gives their LLM a hard coded password because they don't know any better.


Cars didn't exist 150 years ago, yet we do not need the trillion miles of experience that it took Tesla to reach merely a small geofenced region where a steering wheel is not needed; how much would a human need for the same? A thousand miles as a learner? A hundred?

Ten thousand generations of fire and our jaws still have not fully adapted to cooked food, and yet our minds can learn to cope with it just fine; just one demonstration of why not to pour water on boiling oil can be sufficient.

And our genome has to cover our whole body, cell metabolism upwards, not specifically our brains.

There is some stuff our genome gives us, our minds, but it cannot be much. There isn't room for much in our DNA.


Exactly, it's a frustrating bit of theory. The basic observation is that there's a huge weakness in current ML algorithms: humans are about 1 000 000 TIMES more sample efficient than an LLM. Which means it is possible to train and run a model that's in most ways better than ChatGPT ... on a 386, and just millions of tokens (if you don't mind ~15 years training time), so let's say a first-gen i7 in a month.

And it's not just much more efficient in compute, it's obviously not necessary to collect the data in the first place. Which would be another huge saving (the other thing big ML companies imagine is their moat)

Unfortunately people have been constantly trying to do it since at least 1950 and we're still not anywhere close to understanding any of it.

Now where's that French guy that built a 5G base station out of ... I want to say a VCR? We have need of him.


> They are getting much better at the common case. But it feels like we're moving away from AGI and RSI as a tradeoff.

We can but hope.

On the flip side, being good at just maths may solve whatever difference it is that makes all biological brains learn from fewer examples than all machine learning.


> On the flip side, being good at just maths may solve whatever difference it is that makes all biological brains learn from fewer examples than all machine learning.

Will it? Can you help me understand your train of thought here?


> Most of what people call AI slop is literally making AI do design decisions.

I dunno about "most"; the memetic monoculture is also a big part of it, even when it comes up with something that seems novel the first time you encounter it, it's a cliché almost immediately.

While clear patterns in software is not itself a bad thing, this behaviour applies equally to some of the AI's failure modes, and those become instantly-recognisable annoyances.


If every AI were different, they can still learn from each other, without failing together.

This has different failure modes than monocultures, which would also be bad for us: a collection like that behaves as a species, and can evolve faster than one single thing, and evolution is a blind watchmaker that cares only for success rather than details.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: