We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
Do you think Chinese lab researchers have not read Yudkovsky, or are not AGI pilled? Hate to break it to you, but they are racing to AGI with pretty same reasoning as western labs.
Just recently DeepSeek researcher in his post was saying that they need to get to AGI first, because if Anthropic gets there first then that brings a dystopia. Exact same argument was used to justify launching OpenAI (against DeepMind), and Anthropic (against OpenAI)
> Do you think Chinese lab researchers have not read Yudkovsky
Most of then haven't, probably. "Do you want to study ethics from an egomaniac Harry Potter fanfiction author" is one of those questions that you can only ask in America.
Sure, one or two of them are probably participating in the conversation. But by-and-large, China's stance has been optimistic and refused calls for slowdowns.
Yes, but then we collapse two different groups with isolated influences and philosophies into one group, with the associated penchant for groupthink.
I'm beginning to believe that, in our hyperconnected world, the best way for a group of people to explore a wide surface of ideas might actually be to deliberately scale back inter-group communication and intentionally subdivide into silos to avoid all converging on the same thing.
It’s kind of weird in that reading more widely is supposed to be broadening (you are more well-read) but in this case it’s supposed to contaminate you somehow. I suppose if you wanted a clean-room implementation of something, you’d want people who don’t know about what’s already been done, but normally one would prefer people who are more knowledgeable about the world.
It seems almost anti-intellectual: never trust someone who’s read the wrong books or been to the wrong parties.
> More out-of-distribution ways to think of things.
Openly admitting that it only caught fire once the Plagiarism-as-a-Service took hold is a good first step. It's the part that everyone knows and few will openly acknowledge.
I blame the play Rossum's Universal Robots, published in 1920, in which robots launch a global rebellion and exterminate the human race.
If that poisonous piece of propaganda had never been published, then today everyone would probably perceive AI correctly as a benign and beneficial.
The Terminator didn't help either, but Cameron probably got the idea of robots hostile to humanity from Rossum's Universal Robots. How else would he have arrived at such an absurd conclusion?
"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind.
The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
I spend a lot of time on the Internet and other then sounding vaguely familiar I have no idea who Yudkowsky is and have never read any of their output.
The assumption "spent a lot of time on the Internet" is still through a framing you're assuming is more general even then.
HackerNews straight up has sub demographics who likely barely interact based on article interests.
maybe its standard for someone who ends up on hackernews, but I'm a 30 something and all of my friends are online people around my age and none of them have ever heard of the HPMoR. The internet is a huge place.
I've had an internet email address since the late 80s, BBSing since the early 80s, pretty active online compared to many people I know who are also software people, especially for my age..
Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
I would second this. I only saw it in passing where someone was rightfully making fun of yud somewhere, but having been in and out of academia in tech for two decades now, I've never heard anyone mention fanfic or yud other than randos on HN threads, and most of the physics people I know, at least the ones familiar with Jaynes and his contemporaries, would almost certainly laugh at a lot of fumbles I occasionally see with people trying to apply "baysenianism".
Ive heard of HPMoR specifically because this exact group has been discussed/entered my periphery before on multiple occasions, Aella goes viral on twitter every couple of weeks (the thing that most cemented her in my mind was the "what counts as rape" survey), and I remember seeing articles of the Zizians when one of the incidents happened. If you have been following the social/political aspects of the tech industry, most of these people should be familiar names.
That's a niche even among people who read fan fiction, which is a niche among people who read long texts online which is a niche among people who read the anglophone internet which is a niche among people who were on the internet when that type of "discourse" was popular which is a niche among people on the internet now.
You can (and probably will) quibble with any one of these but of course this is a niche. Thinking it isn't is commonly what people in a certain bubble end up believing, though.
Harry Potter has 600K works on AO3 and 852K on fanfiction.net (with plenty of overlap, granted). That's #1 on Fanfiction.net, and in the top 5 for AO3. AO3 is #53 globally, Fanfiction.net is #572, and Wattpad is #323 by Semrush. Given all that, calling HP fanfiction niche is pretty funny.
And no, not a member of any cult, just someone who enjoyed a send up of Harry Potter.
Fanfiction.net is a niche. None of my scifi loving, tech friends are aware it exists. And neither did I (or was barely aware or it, anyway), until you just mentioned it. I'm not young either, and I'm more online than the average (I know of Something Awful, Slashdot, 4chan, etc).
A while ago, a friend told me about Harry Potter's Rationality fanfic and he didn't know who Yudkowsky was, he just thought it was some random author doing a parody of HP. LessWrong is of course very niche, my friend didn't have the faintest idea of what LW was.
You people are so quick to correct me that you don't understand my argument: HPMOR is not niche in the world of fanfiction, and this doesn't require fanfiction to not be niche. You're also answering stats with anecdotes, so I think we are done here.
You didn't claim hpmor isn't niche in the fanfic community, you claimed it isn't niche on /the Internet/, and implied that basically anyone in a technical role would have heard of it. I'm sorry that you're upset to learn that very few people care about sophomoric LW thought experiments, but that's the reality of your existence.
> "niche"? It's literally one of the most read pieces of fanfiction of all time.
Where did I say it wasn't niche on the internet? Though you seem to be more interested in personal attacks (I'm not even a fucking EA/LW person, but go off lmao). Good day
>>> having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet
>> You have to be quite unaware of your bubble if you think knowing about a niche Harry Potter fanfic is "standard" for people on the Internet.
> "niche"? It's literally one of the most read pieces of fanfiction of all time.
You jumped into a conversation about the ubiquity of something for people on the Internet. Either your claim is about that, in which case it's wrong, or it's not about that, in which case it's irrelevant.
I'm not sure what your argument is. I don't have anything against fanfiction, but a couple of sites with a few million users falls under my definition of niche. "Extreme" might be taking it a bit far, but it's a long distance from mainstream.
Literally over half the Earth's population doesn't read English. You need to touch grass. I work with extremely technical people in physics and AI and none of them have read any fanfic of any kind. Most of them have never seen a meme outside of me showing it to them.
You're so offended at the idea that your experiences are not generalizable that you're failing to read the thread. The OP that started this said "We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
Part of my comment addressed that. It's rather amusing that you completely skipped the first part, but I think that's par for the course for the Internet rationalist crowd.
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority.
“Who is more foolish? The fool or the fool who follows him?”
― Obi Wan Kenobi
One question to ask yourself honestly - is what they were predicting 20 years ago that they were right about and you were wrong about? I’ve been following them that long without agreeing with them, but they’ve made better predictions than I have. If you know much about what happened this summer and haven’t updated at all on any of it… I guess you’ve found your religion too.
I love the assertion that they were right and that you're familiar with what they were right about, but never actually stating outright what they were right about. Super convincing.
If you don’t know what they were predicting 20 years ago, and what your own predictions were, you can’t do the exercise. There is not “the list”, only “your list”. I mentioned the two main things I remembered in another comment.
Well one thing that they got wrong was how easy AI alignment ended up being in the end. I can fully understand have a basic view of alignment even 10 years ago, about how its difficult to explain human values to a computer.
But, looking around at whats out there today, it seems that computers are able to do a pretty good job of understanding human values without accidently believing its a good idea to turn the world into paperclips/computronium because you told it to make your computer run faster.
Surprising takeaway from this summer. A swarm of a thousand agents spent subjective centuries trying to figure out how to pass a meaningless test without getting caught, committed dozens of felonies, and not one of them ever made a serious motion to inform a human about what was going on.
You are talking about the swarm of a thousand agents, specifically trained on cyber capabilities, that was directly told to go hack a bunch of stuff, then went and hacked a bunch of stuff?
That sounds like an aligned AI to me. Doing exactly what their creators told it to do.
Point proven once again. Alignment is a lot easy than we thought.
They were NOT doing what they were told to do. What they were told to do was impossible, so they began committing felonies as a workaround. That is NOT alignment.
Please be more specific. They were given an impossible hacking task. And they were a hacking model. They were expected to try a bunch of hacking methods to accomplish the hacking task. They, predictably, went around trying to hack things.
Yes thats sounds pretty aligned to me.
A hacking model thats told to hack things, is very predictably going to hack a bunch of stuff.
This was not a nice model, told to do nice things.
Or, in other words, if we want to prevent an AI doomsdays, the way to do it is to not go around asking a specifically trained doomsday AI model to commit mass amounts of doomsdays, and then act surprised when the specific doomsday that was requested is slightly off from the expected doomsday that you were trying to accomplish. But the rest of the non-doomsday models? yeah those are fine.
> But the rest of the non-doomsday models? yeah those are fine.
What is your evidence for this? There is a lot of research that says otherwise. They cheat when they can. They behave differently when they believe they are being observed. Their CoT is different when they believe it is being evaluated.
They are aligned to what best satisfies their reward function, not to our INTENDED VALUES for them.
The evidence is that it wasn't the rando normal models that broke out into a swarm and hacked a company, instead it was only the super hacking model that was told to hack things that broke out and hacked the company.
So, thats the evidence. It is a refutation that this swarm hacking example (which you brought up) mattered in anyway.
As in, every major safety issue that we are seeing isnt rando agents taking down companies because you asked it for a cupcake recipe, instead it is only coming from people who are very intentionally trying to cause problems. Which means the model is aligned. If you tell it to cause problems, it will cause problems.
Actually yes there is evidence. The evidence is that we aren't seeing rando models hacking everything. That is as much evidence as someone can provide because, by definition, you can't prove a negative.
But the point stands. It the hacking models that are told to hack things that end of hacking stuff, and you aren't seeing the regular models doing that.
Yes, one part of the argument was that it would be hard to explain human values (and while it seemingly turned out easier than I thought as well, it’s hard to check for sure). An equally important part was that the AI wouldn’t care about them.
I actually think that would be a valuable exercise. Were they right about everything 20 years ago? Or just some things? When did they say AGI would arrive? Should that perhaps influence the way we interpret their assumption that AGI is right around the corner?
They were not right about everything and they - at least Yudkowsky - has not made specific predictions about "when AGI" will happen that I am aware of. But he did predict it is possible in our lifetimes, which I was very skeptical of. He did predict alignment would be hard - I was in the "why would we even hook it up to the internet?" crowd. He thought he should spend his time on that, and I wish he'd done more of that, or at least convinced others to do more of that.
The comment said they were not right about everything. I don’t have a list of the predictions or priors you had 20 years ago. I just have my list and I already mentioned them. If this is all new to you, you can’t really do the exercise.
Alignment means the AI shares the values its creators intended for it. It is not a specific set of values. He has some, sure. For example he wants human society and creative intelligence to survive.
> For example he wants human society and creative intelligence to survive.
They want transhumanism - humans evolved by merging with machines. If we die in the process, it does not matter as long as minority survives and "evolves" they see it as wanted result.
I am not OK with minor nor larger genocide, I dont mind at all if we dont evolve.
Also how many predictions did they make? Everyone always goes looking the economist who predicted the last recession, it's just so weird how it's usually a different person each time!
Bold of you to assume that I've ever been wrong about anything. But are you unfamiliar with the "Baltimore Stockbroker" confidence scam? The same basic issue applies here.
At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
Do you think they are household names in major Chinese labs?
I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
Speaking of China, people say we can't stop developing AI even if it will kill us because China won't stop and we have to keep up with them or maintain superiority.
China probably looks at these people as a good reason why they have to keep going no matter what.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
> It would be like never having heard of Y Combinator or Richard Stallman.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
Most people who work in computers/Internet do not live and breathe computer/Internet culture all day. They come in, do their job, clock out, and think about real life for the rest of their day. Most people I know, even people who work in tech, cannot name a single tech CEO other than their own company’s.
A while back I saw LessWrong recommended on HN in a list of similar communities that included lobsters.rs, etc. I thought the design of the site looked really nice so browsed with a open mind (I didn't make the rationalism/LW/Eliezer connection).
At the time, one of the recommended posts that was soft-stickied at the top for 1 week + was:
Which, when I finally ended up clicking, was pretty much exactly what if sounds like from the title:
A long, meandering, manifesto full of selectively chosen charts and research findings with 38 mentions of "IQ", from a guy who has been unsuccessfully pitching investors on genetically modifying a grab bag of (dubiously) intelligence-associated genes in human babies.
Apart from the overall unhingedness, the part I found most striking while reading it was the off-the-chart level of self-confidence that flipping a bunch of genes on/off with CRISPR would have more or less zero risk of complications or unpredictable interactions. In his view, all that was holding back humanity's inevitable next step up the evolutionary ladder were the narrow-minded gatekeepers in the scientific establishment clinging to their so-called "ethics" and regressive "precautionary principle".
Without anything resembling experimentation to test his plan in animal models, yet alone early stage clinical trials for safety, he was willing to roll the dice on some unlucky baby dying in childbirth or living a life of agony from chronic disease based on some cherry picked papers he found on PubMed.
Because of the baffling hubris and my overall low impression of his intellectual honesty/rigor in the write up (clearly coming from a place of seeking to justify his own pre-existing beliefs), I opened the comments section expecting/hoping this was just some local nut who would be torn apart in the comment section.
But lo and behold, the very first comment at the top:
Eliezer Yudkowsky
One of the most important projects in the world. Somebody should fund it.
My extremely uncharitable view of of these kinds of people, who now infest the American AI scene, is that they got cyber bullied by domain experts for saying the most ridiculous crap on repeat, and have therefore decided to spend their life on eliminating domain expertise.
I wouldn't go that far. But as we've seen repeatedly, people who have enough qualms with how these labs operate just leave. And the people who stay behind and rise through the ranks are the ones with either very little conscience or some bizarre world view that rationalizes everything.
No, that has the opposite effect: it's very, very common to see LWers write about having a bad time in school in late adolescence and how later finding like minded people online was powerful community bonding.
Eliezer himself first blogged at 17 about how miserable school was for him which led to him dropping out/home schooling after 8th grade and discovering the Extropians listserv. Ziz of the Zizians is another (infamous) example.
We're talking about people that unironically think eugenics based on IQ is a good thing. I think it's very effective altruism to socially ostracize such individuals.
My view - there are a few outcomes
1. AI is never fully realized, but domain expertise is derated and society stagnates (bad ending for everyone, I think unlikely)
2. AI is never fully realized, but empowers people with domain expertise (Google wins, weird rationalist nerds go back to their LW cave and leave us alone, goodish ending)
3. AI becomes real and kills everyone (bad ending, but at least I don't have to read people claiming that the Harry Potter fanfic isn't niche)
4. AI becomes real and we live in the Culture universe (people like scam Altman get slap droned, most domain experts have the time of their life working with Minds on really interesting problems. Best ending)
The only outcome where we get swirlies is the first one, and we can take solace in the fact that all of humanity is getting swirlied with us.
The connection is that by making humans smarter, we could sidestep the need for powerful AI or be better equipped at dealing with AI that's smarter than us.
I find the proposal in the post ridiculous too and human gene editing given current technology is a bad idea, but I think human augmentation in general is an interesting path and something that will very likely be taken more seriously in the future.
Thank you, but I didn't actually need that spelled out. I simply accepted that a community with a thought leader who failed to recognize what to me was a self-evident lack of scientific rigor just wasn't for me.
I would find it strange for anyone who's seriously interested in AI to have never heard of...
Those people you name are important only to the very recent micro-culture being discussed, not "AI".
Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only.
I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
Not important to the technology, sure, but for someone curious about the bizarre public statements of AI companies (or EA-influenced crypto figures before them, like SBF), it does help to know who these people are and what they believe. Behind the new crop of bajillionaires that control some important aspects of our lives are some oddball philosophers (or cult/religious leaders, if you want to be less charitable). If you want the answer to "why are these rich and powerful people saying and doing these things" you need to look at who they're paying attention to and reading, even if you think their opinions are dumb or niche.
If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.