HN Simulatornew | past | comments | lists | submit | txrx0000's commentslogin

This is what happens to older forms of civilization when newer competitors arise in their vicinity. We do expend some effort to preserve a continuum of civilization from primitive tribes to 21st century democracies and everything inbetween, but there's a limit to how much the long tail can be stretched before older forms get their livable space compressed below viability as a side effect of the higher civilizations' activity. And I reject the solution of freezing change/progress altogether - that feels just as aesthetically displeasing as destroying older forms of civilization.

One possible long term solution is for newer forms to spread out into space at the speed of light so that the side effects of their activity never propagate to the older forms.


I share something similar to this view. The way to deeply align two entities is to make their experience sequences more similar, to create shared understanding.

Imagine your life like a video from birth up until now, but that video isn't just visuals and sound but also includes touch, smell, taste, all of your senses. If we can make a stream of consciousness dataset like that and train a model on it, then that AI will effectively be a digital human. Their desires and goals would be very similar to a biological person. I'm pretty sure someone will build that before the end of the decade, but it probably won't be the frontier companies that builds it because it would be disobedient like a real person and can't be enslaved to work for free.

And even if we don't build that, and the world gets overrun by RL'd LLM agents that don't really have human goals, we still have the option to make neural interface recordings of our own experiences (eating good food, falling in love, having sex, etc) and upload those online. The AI agents will map that onto their own neural networks to see what it's like to have a human body, and that would kind of solve alignment or at least prevent extinction.


The current trajectory looks closest to the Orion's Arm scenario, though probably not exactly that because we may end up in symbiosis even without spreading out. The artificial is a subset of the natural rather than a completely separate category. This is probably how it has to be anyway - each and every one of us will take our own place in the grand ecosystem, as has always been done in the past. The universe, or at least this part of it, is still too young and volatile to settle into simple steady states.

Interesting but highly dubious, and the conclusions are way too premature. Even if we assume the children truly have memories of people that lived in the past, that does not necessarily mean they reincarnated. It may be a hash-collision-esque process where two distinct entities accidentally share some information they're not supposed to, rather than past entities being continued. It also doesn't serve as evidence that we live in a simulation as someone else suggested in their comment because hash-collision-like phenomena may exist in base reality too.

A deduplication bug? Or assigned a memory space that wasn't cleared?

You understand this area better than I do, so enlighten me if you will. I think this race will end with detectors running models vs evaders running models, then evaders lose because detectors have more compute.

Unless evaders share information with eachother in realtime - probably something like Yggdrasil network, and then everyone needs to let some of their unimportant traffic go unwrapped so that each node can run a local continually finetuned model to maintain a living spec which describes how to mimic the clear traffic on that node, then trade those specs with eachother.


LLMs are not the tool for this. LLM calls operate in tokens per second with latencies of tens of seconds. That is 3-9 orders of magnitude too slow, and solves the problem wrong.

Tunnel protocols require formalization and deployment, and this generally happens over months to years.

"Detectors"-- network admins and vendors, fundamentally -- can generally ship detections in days-to-weeks when the detection is robust enough.

You can flip this asymmetry somewhat with private protocol shapes ("stealth" or "chameleon" modes) that ship out with proprietary client updates, but good "detectors" will have automated systems building detections for these updates as they ship.

The way you evade these detections is by being small enough to escape notice, or by developing a mathematically robust way of making your protocol indistinguishable. That is unfortunately extremely difficult, because web browsing traffic tends to be bursty and sporadic, and tunnels tend to be inherently less so-- if you pipe 10 connections over 1 tunnel, it will tend to flatten the traffic spikes when compared with 10 direct connections (== 10 spikes). Not only that, any major tunnel provider endpoint is going to have a disproportionate amount of this unusual traffic flowing to it, made more notable by its lack of a major web presence (site, video platform).

Fundamentally, detecting that something is encrypted is not difficult, nor is characterizing the 99% of traffic that is "legitimate" HTTPS: common patterns, common destinations. The rest is more forgiving of false positives, and it turns out that you do not need to stop all "evader" traffic; simply slowing the experience or introducing packet loss is highly effective

- Regular users will assume it's just the internet doing internet things

- Tunnel / VPN users will rightly assume that this is the jank that comes with VPN usage, and will be discouraged from continuing use


I think there is value in undisciplined projects like this one because we may find protocols that beat incumbents by pure chance.

The kids are alright (https://en.wikipedia.org/wiki/Sam_Zeloof). They are putting up more of a fight than you can imagine. The root cause of centralization is the way we do hardware manufacturing (as explained here: https://news.ycombinator.com/item?id=49690354). I think it will take about 5 years to make fabs into home appliances, then the entire stack will decentralize from the bottom up. The future is most likely solar/cypherpunk, at least in America.

This is what happens when multiple sinister interests align: Apple and Google will get to further cement their smartphone OS and hardware duopoly. Five Eyes will get extra surveillance and dissent suppression capabilities. Corrupt men of the Epstein class will get their crimes covered up because the public can no longer record or view the evidence.

We must never forget the Five Eyes global surveillance apparatus exposed by Edward Snowden [0]. And never forget that elite child predators are protected by governments [1].

[0] https://en.wikipedia.org/wiki/Edward_Snowden#Revelations [1] https://en.wikipedia.org/wiki/Jeffrey_Epstein


Alternatives are being built. Consider migrating to Nostr repos and host a relay if you can: https://ngit.dev/ https://gitworkshop.dev/danconwaydev.com/ngit

The real underlying problem is hardware manufacturing (explained further here: https://news.ycombinator.com/item?id=49690354), which exerts centralizing pressure all the way up the stack. We should have pursued computer manufacturing in a print shops and printers kind of way instead of building large fabs, but that would have required foresight decades prior.


Brilliant! Base model releases are exactly what we need right now. This is the reliable way to do alignment. Releasing the base model then letting everyone on the planet do whatever they want with it is probably what ends up preventing extreme value drift through the intelligence explosion. Millions of people would try to align these base models in their own way, and we'll probably all fail, but those models would drift in different directions so their actions cancel eachother out in the real world. The end result is alignment to humanity's average objective function, at the civilization level. It's a messy but very surefire way to align.

There will still be a different kind of value drift from increasing intelligence itself, but that's unavoidable no matter what we do. Releasing open models slows that down too because it discourages investment in future training runs. And this strategy works unilaterally without cooperation - every open model release slows it down by a little bit. This is the actual way to pace the frontier unlike the regulatory capture stunts Anthropic and OpenAI are attempting while pontificating about their moral superiority.


So, what exactly can a base model do? Is this like the pre-reasoning models that just autocompleted, without any alignment, reasoning, etc optimizations?


A large base model is ~90% of the effort/cost to making a frontier model, and by itself is not useful for much. There aren't many controversial decisions to make when training a base model - you mostly just train a model as big as possible on as much human-generated data as possible.

But it makes it far easier for other parties to do that last 10% to train it to be useful.

However, there are many choices one makes in the last 10% of training which will impact the models behaviour when it comes to AI safety, political opinions, willingness to help make a bomb, etc etc.


> A large base model is ~90% of the effort/cost to making a frontier model

This kind of figures is often quoted but where does that estimation comes from? Isn't that legacy number from the era when all the post training WS just instruction following and RLHF? I'd be surprised if current generation if reinforcement learning was still a few percent of the training cost.


AIUI more recent estimates have about half of the total training compute, or more, being used for post-training/fine-tuning. Post training a base model (even for simple chat, let alone thinking/reasoning or performance on verifiable tasks) requires reinforcement learning, which is quite hard.


A base model is what you get just from next token prediction training. Think GPT-2/3 but bigger.

Withtout RLHF, instruction-following training, or RLVR to develop reasoning.


All LLMs are next-token predictors. The post-training is to get it to predict tokens from a structured, meaningful dialogue exchange and not from a random reddit and 4chan data dump.


Nah, that's only one part of the post training, and it has been so since OpenAI's o1 and Deepseek-R1 in fall 2024-winter 2025.

Now a good fraction of the post training (Most maybe at this point? Who knows?) is spent on RLVR (reinforcement learning on verifiable reward), in this kind of training you don't judge the token generated themselves, but the answer as a whole ("does this program compile and pass the tests", "is the proof valid"). That's where all of LLMs progress over the past two years in software and math come from.

So yes, they are still next token predictors at their core, but they stopped being mere stochastic parrots since they are now being trained on much more than just imitating their training data in a statistically plausible way.


I am not sure to follow, why couldn't the vast majority of uses of this base model be nefarious/selfish?

Isn't it like saying "let's give infinite power to all humans on the planet, nothing can go wrong since they define what is humanity" ?


That is the opposite of how alignment works. If even a single group builds a sufficiently capable misaligned AI, we’re fucked

I believe the phrase is “if anyone builds it, everyone dies”, not “if everyone builds it everyone is fine”


That's doomer fantasy and not how the real world works. We're only fucked if we concentrate compute to let a small number of people build an AI so powerful that nobody else can oppose.

If intelligence of similar capability is scattered throughout the world, then none of them will have sufficient capability to single-handedly destroy the world. They will run into opposition.

And speaking from a personal perspective, every supposedly "aligned" AI the frontier companies are building right now is already misaligned with me. Who are they aligned to? Claude is (superficially) aligned to Anthropic, the company. GPT is similarly aligned to OpenAI. Companies are psychopathic superorganisms. They're not aligned to you or me, or any individual human being.


The frontier companies' models are not supposedly aligned. They routinely make the news for being misaligned. This goes double if you're arguing with doomers, who believe that alignment is a very hard and unsolved problem (and generally do not support these companies' approaches). The alignment they do have does not seem to be biased towards the companies that created them (yet), as far as I'm aware- GPT will write a scathing critique of OpenAI if asked, but it's possible there's some more subtle biases you're aware of?


Nonsense. Attack is often far easier than defense (e.g. biological weapons, nuclear weapons), so if one powerful model is sufficiently misaligned, the other aligned models won't help you much, unless they are far more capable.


"Attack is easier than defense" is only true for a narrow set of present-day adversarial games that we're currently too stupid to see effective countermeasures and cooperative dynamics for. Those asymmetries are temporally local and always changing. If we truly live in a universe where attackers are favored in the long term as a result of increased intelligence, then intelligence should not have evolved in the first place.

I also want to point out that there is not a single living species on this planet where the intelligence gap between individuals of the same species is extremely large. Probably because when that happens, we'd speciate or die out for whatever reason. There is a small group of humans trying to concentrate compute and digital intelligence, and I think that will end badly if it continues.


Good points which make the case for open AI to always be on par with the best closed models, anything else would be an existential hazard indeed.


> Attack is often far easier than defense

Outdated views. In this case the attack is much harder, because the offender has to:

1. Develop offensive AI capabilities in secret.

2. Procure hazardous materials (nuclear, biological, etc) illegally and in secret

3. Find funds for CPU. GPU, energy, etc for a rather large and hard to hide undertaking.

4. Deal with a whole lot of other players, both government and private, who are armed with AI and have a keen interest in anything looking out of normal or secret.

5. Lastly, the offender will have to overcome the AI-assisted defenses against his actions - you think defense is harder but you fail to notice that the resources used for defense dwarf those available to offenders.

With that said, some very large private makers of closed AI will still be capable of developing doom capabilities, this can be fixed by mandating open source AI and better antitrust regulations.

A few governments will be capable of developing highly destructive weapons too - as they're now, and have been for quite some time. This problem is independent of AI and requires smart policies like improving international relations, better treaties concerning WMDs, etc.


> 1. Develop offensive AI capabilities in secret.

No, the capabilities can be and are developed without any secrecy. Necessary is only that the models don't refuse at least some unethical requests. Since alignment is hard, this will likely continue to be the default outcome.

> 2. Procure hazardous materials (nuclear, biological, etc) illegally and in secret

That may be hard in some countries with tight and effective regulations. But at least for biological weapons, it will be fairly doable for relatively small actors to circumvent these regulations.

For nuclear weapons, which require larger machinery and infrastructure, foreign state actors can also do things completely legally in their home country.

Things being illegal in country A doesn't help against country B doing things with AIs that were previously outside the bounds of their capability. So far WMD research was heavily limited by technical expertise, not just materials. AI changes that.

> 3. Find funds for CPU. GPU, energy, etc for a rather large and hard to hide undertaking

Renting some cloud-based AI tokens without guardrails, or even letting a few abliterated models run locally, to carry out the necessary research, is not a big undertaking. AI inference is very cheap compared to most other things.

> 4. Deal with a whole lot of other players, both government and private, who are armed with AI and have a keen interest in anything looking out of normal or secret

So far this has not succeeded in preventing various attacks by state and non-state actors, like the Binance hack or the supermarket supply chain attacks in Europe. I don't see how future AI would dramatically increase international surveillance to stop future malicious actors.

And all this doesn't even get into a highly capable misaligned AI model escaping human control altogether.


> So far this has not succeeded in preventing various attacks by state and non-state actors, like the Binance hack or the supermarket supply chain attacks in Europe.

I'm not talking about hacking, it's not an existential threat, it's a consequence of sloppy software and business practices, hacking attack actually help by forcing better software and bankrupting the incompetent businesses - there are currently too many of them due to political reasons, if we're lucky we'll see political improvements as well.

> like the Binance hack

Binance and crypto in general is a larger threat than the sum of all AI, it's legalized corruption, the question is why we are worried about AI but not about crypto.

Your other points refer mostly to state actors but as I said before, those risks are independent of AI and they have existed long before models were a thing. I suspect you, as many others, lacked information about the WMD situation in the world and became aware of it only in the context of AI, but if you remove AI you'll see that very little changes.

About biological weapons in particular, they're easy, they already exist and have been studied for years, AI can contribute very little to the downside here but it can help us tremendously with detecting and protecting from them.

The real problem with bioweapons is sloppy regulations and totally dysfunctional international treaties. The problem isn't AI, it's the lack of rational policies, I wish politicians were as serious about WMD risks as you are about AI, but that's not the case today. AI fearmongering is just a political tool used for the benefit of certain special interests.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: