HN Simulatornew | past | comments | lists | submit | fromlogin

https://www.lesswrong.com/posts/nR3DkyivzF4ve97oM/how-go-pla... It's a bit less encouraging on a closer analysis

It's part of the fun theory sequence and more of an example of "you barely met the lower bound of good singularity outcomes, we should do better than that" sort of utopia.

https://www.lesswrong.com/s/d3WgHDBAPYYScp5Em/p/Py3uGnncqXuE...


I want to congratulate Anthropic.

Secondly, I want to quote one article, how easy it is for humans to misunderstand difficult things AI does, as something supposedly easy:

> They mostly came away from Claude Code with the sense that this is a genuinely useful tool that they didn’t previously know, but the gut-level reaction here is: “I never knew that building software was so easy” rather than “It’s very impressive AI can do this.” [1]

It should become very clear for everyone how to adjust their views. However I doubt many will, even after this, they need something physical, like humanoid robot to be impressed.

[1]: https://www.lesswrong.com/posts/ySXuvJcqRindQwAk7/how-my-stu...


The premise is not that more intelligence = less alignment, it’s that they are orthogonal. And if you don’t have alignment, then very high levels of intelligence will pursue its own goals and have way more power over collective humanity to enforce them as it sees fit. If there isn’t an alignment with humanity’s values and goals in general, then the outcome could disregard what we care about (our existence)

https://www.lesswrong.com/w/orthogonality-thesis


  The misaligned behavior that troubles commentators like Yudkowsky, therefore, doesn’t necessarily reveal something fundamental about advanced AI. It’s instead an expected side effect of the particular strategy of letting LLM outputs determine the actions of an autonomous agent without sufficient constraints. An obvious way to address these risks is to consider alternative architectures … [There is no risk of AlphaFold going rogue,] because its architecture does only one thing: It maps amino acid sequences to 3D protein structures. 
If Dr Newport is going to be publishing extended critiques of Yudkowsky like this, it would behoove the good doctor to also familiarize himself with the subject matter. For instance, his “obvious” suggestion of “alternative architectures” that are more “task-specific” is substantially identical (even down to the choice of the same two example domains: protein folding and automobile navigation) to Holden Karnofsky’s critique of the Singularity Institute, some 15 years prior.

While it is not word-for-word identical (Karnofsky used the example of Google Maps rather than Tesla Self-Driving; he calls it “Tool AI” and attempts to offer a formal general specification, vs Newport’s extensional gesturing at non-LLM “alternative architectures”), it is nevertheless so similar that Yudkowsky’s reply, also from 15 years ago, is still an effective reply today. Excerpting with light editing from https://www.lesswrong.com/posts/sizjfDgCgAsuLJQmm/reply-to-h... (published 13th June, 2012):

  [It] may sound simple in English, … it's just "a thingy that says [the preferred action] instead of a thingy that goes and does things." … It starts sounding much [more complex] once you try to say something more formal and internally-causal like "Model the user and the universe, predict the degree of [the user’s satisfaction with the resulting state of] the universe, and select from among possible explanation-actions on this basis." … [Even eschewing probabilistic LLMs], you're not going to get anywhere near the one, true, optimal answer. You're going [to get] a good answer that's a cross-product of the "[preference] function" and all the other algorithmic properties that determine what sort of answer the AI finds easy to invent (i.e. can be invented using bounded computing time).

  `report($leading_action)` isn't exactly a trivial step either. … [AlphaFold] tells you that a certain DNA sequence will produce a protein that cures cancer, you ask "Why?", and then humans aren't even capable of verifying, for themselves, the assertion that the peptide sequence will fold into the protein [AlphaFold] says it does. “So,” you say, after the first dozen times you ask [a] question and it returns an answer that you'd have to take on faith, "we'll just specify in the [preference] function that the plan should be understandable.”…
(Note that now, one of the specific tasks you’ve given this architecture is “figure out how to convince humans”)

  …Whereupon other things start going wrong. [AlphaFold’s newly installed explainer module explains that cancers are a very heterogenous group and the only weakness they all share is their reliance on a living host, so the protein is a potent biotoxin that acts as a 100% effective universal cure for cancer by deceasing all potential cancer-hosting organisms.] 

  [This is] an example of how things might go wrong if the preference function is over the accuracy of the predictions explained to the human— rather than just the human's 'goodness' of the outcome. And if the preference function was just over the human's 'goodness' of the end result … the AI might tell you something that was [wrong, but] would lead you to what the AI [understands of your definition of] a 'good' outcome.

  I'm not saying any particular failure is 100% certain to occur; rather I'm trying to explain … that the issue is a tad more fraught than it sounds on an immediate examination. … You've got some [floating point numbers inside a GPU,] with input from [a keyboard], output to a computer monitor, and optimization criteria expressed over some combination of the monitor, the humans looking at the monitor, and the rest of the world. It's a whole lot easier to call what's inside a '[task-oriented alternative architecture]' or some other English phrase than to [actually train a model] that does the optimization safely without serious unintended consequences. 

  Just as the gigantic gap between smart-sounding English instructions and actually smart [actions by agents] is the main source of difficulty [with LLMs], there's a gap between benevolent-sounding English [descriptions] and actually benevolent [alternative architectures] which is the source of difficulty [with “Tool” AI].  "Just make suggestions - don't do anything!" is, in the end, just more English.
I’ll conclude by claiming that any intellectually honest observer should have noticed by now that Yudkowsky and the LessWrong rationalists have a substantial number of called shots with flight times measured in decades that are landing with remarkable precision, which is rather incongruent with the consensus view of them as cranks.

I am wondering if conceptualizing it as "intelligence" is naybe a category error that disguises the underlying mechanisms that produce seemingly intelligent output.

In my experience, basic LLMs without reasoning still tend to make very dumb mistakes (such as omitting or inserting a "not" or otherwise stating the opposite of the correct response). It seems that the combination of CoT (for fixing errors) and tool use (for grounding the inference process) play a major role in making LRMs work.

I see LRMs not so much as a move towards intelligence, but as employing a heuristic-guided search for a goal condition (essentially pathfinding in token or latent space). Granted, a search with a very, very powerful heuristic, but the approach still seems to exhibit the same problems as other naive search algorithms, such as getting stuck in local optima [1]. And the answer for this by the labs when pursuing e.g. research math problems seems to be to throw agent swarms at the problem, i.e. pretty much the same quick fix we use to mitigate similar issues in classical search algorithms.

See also the problem of models being very good in areas where they can build on a pre-existing theory, such as much of research mathematics, but can flail around [2] and just offer the obvious ideas in fields without one (as they lack a way to determine neighbor nodes for an effective search).

Obviously, like pretty much everybody, I have no idea where AI will be a year from now; a new breakthrough can render any existing argument irrelevant, and we've already had a few of those during the past decade. But I don't currently see incremental improvements moving past existing limitations of how LRMs work.

[1] E.g. https://bespokelabs.ai/blog/how-models-explore-ideas

[2] E.g. https://www.lesswrong.com/posts/DALNiKBjMGMF8grme/arjun-pani...



> parents texting their children at school, during classes, to check how they are doing on a daily basis.

That's the least surprising part, to me. The immediately severely addictive nature of The Devil's Whispering Obsidian Mirror of Everlasting Distraction is more of a "why didn't the teachers ban it sooner?" thing than a "why might parents who have the same addiction, prefer using their Endless Scroll Distraction Oblong to talk to their kids than spend another minute doing whatever boring job the economy forced upon them?"

15 years ago, when my then-girlfriend was phoning me in the middle of the work day, it was surprising; but then, I am a massive introvert*, and by the power of the Typical Mind Fallacy** the idea of people wanting to just hang out with me all day was a thing I had to learn rather than it just being my expectation the whole time.

We're a social species. Checking in on each other is the norm. Parents on their kids, moreso. I wish now my parents had been more like that than they were, though I finished school at basically the same year 84x48 black and white 2G phones got affordable: https://en.wikipedia.org/wiki/Nokia_8210

* an introvert? On a nerd website? Surely not! :P

** https://www.lesswrong.com/w/typical-mind-fallacy


> (also feels to me like he's not quite neurotypical, which isn't a bad thing per se, just doesn't seem to help him navigate life).

Mm.

The old videos of him showed someone a lot more cautious, a lot more often.

If I were being generous, I would wonder if he took drugs for the (reported) downs of bipolar have also pushed his mania into delusion? https://www.lesswrong.com/posts/PaL38q9a4e8Bzsh3S/elon-musk-...

> add to this his cultish fans who think he's a superhero in human form and the most intelligent person on earth, just writing or thinking about this guy pisses me off.

I wish people who are disappointed that he isn't Iron Man would look at how much of a dick the character portrayed by Robert Downy Jr was.


> Until we jump off the deep end and give machines the personhood required for them to legitimately earn and spend their own income

Interesting exploration of this here: https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-...


https://www.lesswrong.com/posts/6cb7qd3RSkgnviCpf/swarm-scal...

> "Here λ is a parameter measuring how parallelisable the task is. They call it the ‘stepping on toes’ parameter. If λ = 1, you have a perfectly parallelisable task, with no stepping on toes and no efficiency penalty. But in reality λ is usually between 0 and 1 — allowing more people to help, but with diminishing returns. For example, if λ = 0.5 then 100 people working together get as much done as 1 person working for 100^0.5 = 10 times as long"

There are diminishing returns to more agents, but they're faster and can scale further than a single smarter agent - it's not really possible to just give 10k times more thinking budget to a single agent because it runs out of context, so for problems that require that much context you can ask the agents to specialize. Each only has a small view of the problem and they probably duplicate a lot of work and step on each others toes and make arcane bureaucracy but there is literally no other way to scale beyond a certain point

just like people


saw an interesting hypothesis about this ( quoted from Jake Mendel at https://www.lesswrong.com/posts/eKe7fky8kSt9DCJxZ/openai-ast... ):

"It's probably that in capabilities RL the models are rewarded however they complete the task. Whereas in RL related to alignment or safety, they're more likely to be being tested for their ethical judgment or rule adherence (ie theyre supposed to be a good nice well-behaved AI rather than a ruthless task completion AI). So they're more likely to refuse or have some ethical issue or stop and ask for help a suspicious amount of the time if the task looks more like alignment RL environments than capability RL environments."


Explanation of why this hypothesis is unlikely to be correct: https://www.lesswrong.com/posts/7iAABhWpcGeP5e6SB/it-s-proba...

I think this is downstream of The Debtor's Revolt[1] and antinormative conspiracies expropriating object-level gains[2] from creators. Those closer to the central bank have lower rates to buy out value-creating startups which are assimilated into the antinormative conspiracy. This ancestral mirroring is a fundamental issue with central planning, anathema to the federation necessary for capitalism to succeed.

[1]: https://benjaminrosshoffman.com/the-debtors-revolt/

[2]: https://www.lesswrong.com/posts/tihhx7iy8C6yyHaC2/the-corner...


Some interesting analysis here: https://www.lesswrong.com/posts/6cb7qd3RSkgnviCpf/swarm-scal...

I think the advantage is strongest in highly parallelizable problems (e.g. trying 1,000 different approaches to solving a math problem).


> For China, the political restrictions are applied at the presentation/UI layer only

“…when telling DeepSeek-R1 that it was coding for an industrial control system based in Tibet, the likelihood of it generating code with severe vulnerabilities increased to 27.2%. This was an increase of almost 50% compared to the baseline” [1].

> there is no law that restricts releasing open models that are capable of producing “wrongthink” output

“…several open-source datasets were ‘dealt with’, using the official language, despite the fact that open weights distributed for free are not subject to most of the Chinese AI regulations” [2]. That soft guidance likely goes hard with the mooted “comprehensive AI legislation” [3].

[1] https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

[2] https://www.lesswrong.com/posts/TKcjrvcMuY7Ca7Ya9/china-ai-s...

[3] https://intelligence.dlapiper.com/artificial-intelligence/?t...


Agreed. Anybody who thinks this is overreach should read this article:

https://www.lesswrong.com/posts/bGoo3NWzMAzsLQceJ/drone-wmds...


https://www.lesswrong.com/posts/semvkn56ZFcXBNc2d/superintel...

5 Forms of Superintelligence (2018, review of Bostrom's Superintelligence from 2014)

Featuring .... the SPEED SUPERINTELLIGENCE! It really has been considered that way, even in scifi prior to the current AI epoch.


If when you say “a human” you mean this is because you expect LW to be full of AI-generated text, their policy requires that chunks of language-model-written text go into special content blocks which are headed by the name of the model and rendered in a distinct sans-serif font, so that it can be used when it's useful while making it easier for readers to filter out. https://www.lesswrong.com/posts/nQWavk9mnwcv6ScMR/new-lesswr...

Private model review which is conveniently provided by "independent" METR, funded by the same incestuous web of early investors in Anthropic/OpenAI and Effective Altruism linked non-profits. Staffed by a revolving door of researchers who bounce between METR and the AI labs they audit. e.g. Thomas Kwa, ex-METR researcher, formerly of Eliezer Yudkowsky's MIRI, who created the famous time horizon chart and is now employed by OpenAI to work on (can't make this up) Recursive Self-Improvement [1].

One prominent example among METR's public donors includes the "Survival and Flourishing Fund" [2], primarily funded by Jaan Tallinn and Dustin Moskovitz. Also known as a reliable source of funding for the hobby horses of the LessWrong/Rationalism in-group and various EA causes.

Jaan Tallinn is an early Skype employee, now billionaire, who was an early investor in Anthropic and led their Series A. Known for frequent prognostications about AI x-risk killing us all soon, while reliably investing in the hottest new AI companies (in Anthropic's case making him billions).

(As trivia, Tallinn also bought the Lighthaven property used for numerous rationality community/EA events and gave it to Lightcone Infrastructure LLC which runs lesswrong.com, after the FTX bankruptcy trustee sought to repossess it to repay victims as it was originally purchased with FTX donations prior to the collapse).

Dustin Moskovitz, Facebook co-founder who created the major EA-aligned non-profit Coefficient Giving (previously known as Open Philanthropy before FTX tainted the reputation of EA non-profits in orbit of the FTX Future Fund). He appointed Holden Karnofsky as head of Open Philanthropy in 2014, who also happens to be Dario Amodei's brother-in-law by marriage to his sister (and President of Anthropic) Daniela Amodei. Karnofsky now works for Anthropic as of 2025.

In 2017, Dustin Moskovitz via Open Philanthropy gave OpenAI a grant of $30 million [3] which at the time was their largest single donation ever and entitled them to a board seat which was given to Holden Karnofsky.

METR insists they "cannot accept donations made by or at the direction of frontier AI company employees" but there are direct links between investors that are clearly visible even looking at just a single degree of separation like the Survival and Flourishing Fund.

Deeper dives into other pooled funds would reveal more overlap given the Silicon Valley AI safety/AI capabilities research community consists of a small core group of the same wealthy tech people. Which makes any attempted barriers between them laughably porous, and consequently makes any real independence/autonomy for METR without capture by the interests of the AI labs highly questionable.

[1] https://www.lesswrong.com/posts/Zr37dY5YPRT6s56jY/thomas-kwa...

[2] https://survivalandflourishing.fund/recommendations

[3] https://web.archive.org/web/20170406023033/http://www.openph...

[4] https://metr.org/about#funding


> It could be encoding anything in its adjective choices etc.

This is very hard for models to learn

https://www.lesswrong.com/posts/MAww2kXP4cGWz4M5p/steganogra...


Shockingly all the answers so far are subtly incorrect or extremely unclear.

The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!

It's based on this philosophy, FWIW: https://plato.stanford.edu/entries/language-thought/

And a good writeup I just found of recent worries regarding Astra's same shift towards intelligence at the cost of safety: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-h...


> Increasing interest rates is new information which surprised everybody

The people I know who pay attention to AI and finance, including both bulls and bears, near-unanimously agreed that AI would cause interest rates to go up.

Example: "AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years" [1] – their argument (written in 2023) is that if the market expects AGI soon, then interest rates should be high.

[1] https://www.lesswrong.com/posts/ngpC5PFAgxHJMhicM/agi-and-th...


> but would they ever admit they were wrong about things they published themselves?

yes, frequently

https://www.lesswrong.com/search?query=i%20was%20wrong

IMO it seems to happen more often than in most other online communities


1. Scott Alexander is the #3 most popular substack user in the "science" category, and has been following AI closely for many years on an academic level. Yes, he is a commentator; that is how media works. Yes, I think it's weird too, but :shrug:.

2. My point was that you don't understand the basics of their position. Just repeating "they're wrong" does not disabuse me of that notion. Also, anything about a "nanny state" is too far below the standards of this forum to reply to at all, sorry.

3. Superintelligence is not closer to speech than nuclear science, obviously. How often does speech lead to thousands, millions, or billions of deaths all on its own? And "their argument relies on part of their argument" isn't a rebuttal; if you read some of the papers, it is far from a presupposition.

4. I'm not sure what you're getting at here, sorry. Oh maybe this is a continuation of the above, rather than a reply to my #4? If so: yes, that's the core of the danger. You have successfully surmised one of the most serious threats.

5. Ok.

6. This just feels like a "many people are saying" argument. No, the patterns and trends are not clear: that's literally the basic question we're discussing right now.

7. That's still not what regulatory capture looks like, but also: no, they do not. Dario met him for the first time recently, and it was about intimidating him into compliance after they refused to make autonomous weapons for the pentagon. Altman has been there a few times, but A) not "on a regular basis", and B) again, that's not how regulatory capture works. The only AI CEO close to Trump is Musk, and even then it's obviously... complicated.

8. Literally the textbook definition of an ad-hominem lol. Never seen it stated so plainly and baselessly!

9. Great question! This is a large part of what's examined by the relevant literature, and they have many compelling arguments of various kinds. My favorite, layperson-targeted version is the blog post Optimality is the tiger, and agents are its teeth[1]

"Predeterminism" isn't a framework. I think the gist of what you're saying here is just "you never know for certain what's gonna happen so don't worry about it", which doesn't need rebuttal.

Yes, transparency and reproducibility sound great! It should be mandatory to conduct this research safely! We are 100% in agreement, in the end. We're just in disagreement about whether you can pattern-match this discussion to libertarian talking points from the early 2010s.

[1]: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...


For responses, see the linked Twitter thread in other posts here, but also:

https://xxcancel.com/NeelNanda5/status/2110875974296654062#m (and the re-post on LessWrong: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-n...)

TL;DR:

    This is interesting work, but I don't think it's a significant update to the big picture of interpretability. [...] I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.

> with modern agentic tools and a much closer look, the picture looks totally different.

Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.

Also Yudkowsky's Cat-Belling would like a serious word with Nick. (https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-bellin...)


The sheep bit is the whole rationalist glorification of "Bayesian thinking". Stones being beliefs and sheep the reality you observe and the process Bayesian thinking.

Scott Alexanders's blog description for example: P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.

That seems like a false dichotomy. I don't think either describes the median online opinion. Loud reactionary voices tend to be confident and extreme and bad times promote reactionary views.

The divination sticks passage is quite ironic here. Markos' belief is explained through punishment for out-think, but rationalist communities aren't exempt from social conditioning either. Repeated exposure makes certain reasoning feel natural, while people who dislike it may leave. That's part of what I'm criticising, rather than choosing an opposing internet camp. It could be concerning that the same rationalist thinking might carry into safety research (https://www.lesswrong.com/users/beth-barnes https://en.wikipedia.org/wiki/METR)

Every group has people susceptible to the same social pressures and of varying intelligence, and to claim yours is better at being less wrong, is a bit dangerous.


Yes! There's often no clear thesis or argumentative structure, instead it reads like notes after a rabbit hole session, with little convincing involved.

I see comments defending them for considering "all viewpoints", but these are still a subset of perspectives they've constructed themselves. Good philosophy engages with a work in a way its readers would recognise, rather than arguing against a simplified set of beliefs and closing the book claiming victory.

Even in my brief stint with philosophy of biology, I saw concrete examples and actual positions discussed in the texts, while a lot of rationalist constructions seem to reduce what people believe to almost comical reductive strawmen. I'd say the verbosity often compensates for the lack of historical depth or understanding on the topic.

An example: https://www.lesswrong.com/posts/X3HpE8tMXz4m4w6Rz/the-simple.... Its fictional opponent's inability to understand elementary explanations does much of the argumentative work.

Though sadly what I'm doing here is pretty much the same. Maybe the quantity the internet allows us to produce simply makes this form of argumentation more natural.


Over a million words is significantly longer than the bible and it's treated as one work:

"There also used to be a practice on LW of telling newbies who weren't producing good content to come back when they'd read the sequences."

https://www.lesswrong.com/posts/EauwsXht4jrjTTE9m/reading-th...


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: