As far as manifestos go HPMOR is a pretty good one. I think that’s what a lot of the critiques about it miss. Yudkowsky is actually a pretty decent writer, especially in the fan fic scene. Not only is his prose actually acceptable which is pretty rare in fanfic it’s clear that he actually thought through a lot of the plot points ahead of time. Harry Potter universe fits very well for his story not only because it’s popular with the target audience, but because there’s a ton of unexplored authority figures in the original and the main goal of the manifesto largely boils down to rejecting blind belief in authority and blazing your own trail. I think it’s hard to separate the number of people who read it because it’s legitimately the best Harry Potter fanfic from the people who read it for the manifesto stuff which again points to the ingenuity of choosing it as a setting.
For me this is probably the weirdest part of it. I have read HPMOR and was distinctly underwhelmed by its literary quality. I mean, it's not bad, but it's not serious literature. It's not even on the level of the original HP. It's a fanfic. Sure, it's a fanfic with a lot of fan service and geekery and so on, so the audience is targeted masterfully. But still, it's hard not to notice the numerous literary faults. I mean, "he actually thought through a lot of the plot points ahead of time" is a kind of praise, but not one that a decent literary work expects. That's like saying about a chess grandmaster that he actually knows how all the pieces move. I mean, sure, on the fanfic scene it's probably above median, but I've seen people making out of it some kind of epic revelation, akin to some kind of a sacred text. Completely undeservedly, in my opinion, and rather weird for a community that is so proud in it's objective and rational approach to things that put it into its very name.
It's terrible as a work of literature. It has many, many, many flaws, but the most serious is that it's boooooring. Hundreds upon hundreds of pages go by with nothing happening except creepy children preaching at each other.
It's terrible as a fan fiction. Not a single character acts anything like the original, the world is unrecognizable, and the plot is derailed entirely by the end of the second or third chapter. The original story needs to be distorted beyond recognition because the author's bizarre worldview works even more poorly in a fantasy world than it does in reality.
It's demonstrably effective as a manifesto, but that is despite the fact that it is riddled with scientific and conceptual errors.
It's one of the worst things I've ever read. The fact that people like it is evidence of the bizarre incestuous nature of the community that produced it.
Anyone who knows anything about Harry Potter fandom in those years knows that its readers were always thirsty for something new in the Potterverse to read or be exposed to. By latching onto that universe's incredible popularity, Yudkowsky exposed a large number of young, inquisitive people to the ideas of rationalism. It was essentially a recruitment mechanism, one that was wildly successful. Many rationalists freely admit that their first exposure to rationalism was via HPMOR.
I don't think Harry Potter fans would look kindly on their favorite story being treated with such contempt. Remember that within about three sentences of introducing Ron Weasley our hero finds him guilty of the heinous crime of enjoying sports and concludes he should not exist.
Excuse me. It was Draco Malfoy, our rapist deuteragonist, who said that. Our hero merely agreed.
Maybe Harry Potter fans would read a bad fantasy book, but HPMOR is not just a bad fantasy book. It is repugnant to healthy people from the first few pages.
I really, really dont understand this complaint at all. Harry and draco disliking ron is supposed to show that they are snotty kids who have big blind spots. The characters are supposed to be dislikable at the start, the story is about the growth they experience from there. This is part of what makes it a better story than the original. Yudkowsky's harry grows as a character which basically doesnt happen at all in the original series.
Do you dislike succession too? Some people just need completely heroic protagonists in their stories. Thats fine, but it doesnt make the story bad.
They are introduced as insane, not snotty. Nearly the first thing Draco says is that he plans to rape Hermione and our Hero uses his Agency to do exactly nothing about it. (It is hard to exaggerate how bizarre this story is to people who haven't read it.)
Hundreds of pages later, Harry uses the first iota of power he is given to prohibit students from playing Quidditch according to rules he believes are irrational. There is an immediate outcry which he dismisses because it's just the mewling of inferior intellects. It's hard for me to see any signs of growth here.
(Harry's contempt for sports is amusing, given that about a third of the story is dedicated to his love of cheating at a particularly moronic sport invented by the Lord Voldemort ripping off Ender's Game.)
He convinces Draco that his whole ideology is wrong. He convinces him not to rape anyone before he rapes anyone. Not exactly sure what the better outcome is here, just turning him in would only strengthen his violent ideology. There are many complaints to be made about the story, as you said its boring and too long and often too preachy. A lot of the visible changes in the characters happens in the last quarter. Even though the whole story has been building to that it can seem like its a bit out of nowhere.
Does Yudkowsky spill more ink on condemning Draco's hatred and violence or on making fun of Hermione Granger's feminism? Voldemort is treated as an ultimately worthy adversary, while Ron is a joke and Hermione is a pet handed power by our Awakened Hero. Yudkowsky treats sexual violence, genocide, and mass murder as forgivable missteps on the road to greatness, but treats liking sports as a fatal character flaw and being concerned with human rights as risible. Arrogance is consistently rewarded while goodness is consistently ridiculed.
I don't claim Yudkowsky believes these things consciously, but they are omnipresent in his story and they reflect his bizarre attitudes to morality. The fact that he acknowledges at the very end that the characters he created are fools doesn't change the fact that he spent six hundred thousand words holding them up as paragons of virtue and rewards them with apotheosis.
Hermione is the actual hero of the story. Every time rationalist harry makes a mistake he is acting against her direct advice. Her character literally exists to show off his deficiencies and his coming of age basically boils down to realizing he should be more like her.
For many reasons voldemort is an ultimately worthy adversary. Much of that has to do with Harry's biases that come from being too invested in rationalism. Since he considers himself above everyone he's ever met he longs to find someone he considers his better and voldemort fills this role. He wants a mentor so badly that he's willing to throw out his entire ideology to keep him. I think yudkowsky's portrayal of voldemort as the ultimate adversary to rationalism is better than pretty much any plot arc in the original story. Harry does ultimately completely reject voldemort so Im not sure how sexual violence, genocide, and mass murder as treated as forgivable. From the very beginning Harry is deeply concerned with human rights, his distractions and the path he takes toward achieving human rights in this messed up world is again an integral part of the story.
> The fact that he acknowledges at the very end that the characters he created are fools
I think this is my biggest issue with your characterization. Yudkowsky shows harry to be a fool from the very beginning, as you pointed out with the draco/ron stuff. With his dismissal of much of hermiones advice. He very consciously did not create harry as some ubermencsch character who can do no wrong. He shows that harry believes himself to be a parity of virtue, but that he very much is not until hermione's death.
This is a forgivable statement if you didn't read it, because then I guess you may not have seen that there's themes at play and a point being made. If you did read it I do not understand how you don't see how you are meant to dislike the summary judgement and the judgement itself being an effect of yet another plot point.
But then if you did not read it it seems a little disingenuous to say "this is a bad book", because you would not only not know but anything you did know you would not understand.
What is there in the story that indicates this judgment is wrong? Ron does not appear in the story again, while by the end Harry is an immortal superbeing. His arrogance is never once punished.
You can be a sexually predatory sociopath and still be of value in Yudkowsky's world, but like sports and you're dismissed in your entirely within a single paragraph.
> "Ah..." Harry said. This was very hard to say. "Actually, the rest of the world doesn't know this part, not even the Headmistress. But in point of fact you were one hundred and twelve percent correct about [spoiler], and I'll remember for future reference that although 'Dark' and 'evil' may not technically be the same thing, there's a great big statistical correlation."
> "Oh," said Hermione, and fell silent again.
> "You're not saying that you told me so?" said Harry. His mental model of Hermione was yelling: I TOLD YOU SO! DIDN'T I TELL YOU SO, MR. POTTER? DIDN'T I TELL YOU? [spoiler], I SAID, BUT YOU DIDN'T LISTEN TO ME!
It feels like most people who criticize the story didn't actually read it, and just start with the premise that Yudkowsky sucks at writing, then conclude that obviously Harry Potter must be Mary Sue, and that whatever Harry says is what Yudkowsky wants you to believe.
If you actually read the story, you will find out that many things that Harry said and believed turned out to be wrong. That ending was written by Yudkowsky, too. The characters that disagreed with Harry since the beginning, such as Hermione and Dumbledore, were written by Yudkowsky, too. Perhaps there is a little more depth in the story than if you assume that it must be a Mary Sue story.
Ah, yes, the ending where the invincible Superman uses his gigabrain to singlehandedly save humanity, end death itself, get the girl (and turn her into an immortal magical unicorn princess (yes, really, this story is dumb)), vanquish a dozen Wizard Hitlers at once, and tear apart entire stars for their raw resources shows that he paid the price for his earlier arrogance.
I have read the story in its entirety. Don't ask me why. The fact that our Lord and Savior holds the idiot ball on occasion does not make him realistically flawed. The fact that Voldemort would have realistically been exposed within thirty seconds makes the story dumber, not smarter.
Also, you ignored the question I actually asked: where does Harry realize that it is wrong to steamroll everyone he thinks is stupid?
> It's terrible as a work of literature. It has many, many, many flaws
Pretty much all fan fiction is terrible literature. I find yudkowsky's prose to be above average for fan fic which admittedly is just saying hes among the best of the worst. Every single fanfic ever written that doesnt end after 1 book has significant boring sections, theres no editor.
> Not a single character acts anything like the original
This is the point. Yudkowsky recognized that the hp world acts as a natural foil for his ideology. By changing the characters he points out the original characters flaws, but also the ways they are better than the rationalist versions of the characters. I dont understand why people claim fanfic should be always be faithful to the original setting. Using it as a foil can work too. The characters in the original, especially the authority figures are all incredibly flat. Why would you want more of that? Even the ones he largely keeps the same, like dumbledore, he explores in a new light which makes them seem different even though they are very similar to the original. The reveal that the 2 Dumbledore's are actually very similar(the exact same?) characters is central to understanding rationalist harry's deficiencies.
> The fact that people like it is evidence of the bizarre incestuous nature of the community that produced it.
Agreed but I think this says more about how crap HP fan fic tends to be than anything else. Yudkowsky saw unmet demand and used it to shoehorn his ideology in
Let me be clear, he's no cormac mccarthy(my favorite prose is the passage from blood meridian describing the raiders). My bar for fan fiction prose is basically whether it annoys me, its a low bar. Most of what makes hpmor good is the cleverness with which yudkowsky subverts the original harry potter world to try to make a rationalist argument, whether that be by inverting a character or playing off the harry/voldemort relationship in thoughtful ways.
'"Voldemort," said the old wizard. "I understand him now at last. Because to believe that the world is truly like that, you must believe there is no justice in it, that it is woven of darkness at its core. I asked you why he became a monster, and you could give no reason. And if I could ask him, I suppose, his answer would be: Why not?
They stood there gazing into each other's eyes, the old wizard in his robes, and the young boy with the lightning-bolt scar on his forehead.
"Tell me, Harry," said the old wizard, "will you become a monster?"
"No," said the boy, an iron certainty in his voice.
"Why not?" said the old wizard.'
as you can see its nothing special, just good enough
I see, thanks. It reminds me a little of those spam emails which are sprinkled, intentionally, with obvious spelling errors. They aren't trying to trick the average person, they are trying to filter for a much smaller, more valuable audience.
fwiw, I don't think that was intentional here. He's just more interested in "ideas" than prose or people. A more extreme example of Andy Weir. Definitely a niche. I read it years ago while backpacking and it perhaps needed an editor but the overall struggle/war and reimagining of logical fallacies of the HP universe's magic were fun.
yes exactly. Fan fiction readers largely do not care about grammar or nice prose, just has to not be overly distracting which unfortunately most fan fic fails at
fantastic. I've heard of bits and pieces of this but this is the first article I've read which lays out the entire timeline with all the firsthand sources you could want. thanks for the effort!
> His autobiography even opens with a parable: as a small child he threw his favorite squirtgun off a third-floor balcony rather than accept that his mother could lend out what was his, and felt complete when he heard it hit the pavement.
What a classic narcissist.
> Main character syndrome stops being a character flaw the day you join a religion whose plot has you in the lead.
I've tried to read this, but somehow the article doesn't really come to a point. Is there any point besides Yud not being perfect, and having connections to people who openly talk about their sex life (and that it's somewhat non-standard)?
I think the point is that Yud and the rationalist community have put themselves in an awkward position where their lifestyle now depends on "p(doom) is big" being true. Which means they can't necessarily be trusted to give unbiased updates on the true level of risk we're facing as new data and information comes up.
Can "ASI will inevitably kill us all" be falsifiable before we create ASI? If not, then what separates it from religion?
Not sure how much I agree with that position, but that's just how I understood it.
> Can "ASI will inevitably kill us all" be falsifiable before we create ASI?
No, but there are decent proxies, like "Alice who is X standard deviations smarter than Bob will win >50% of the time at game Y even given a disadvantage that would make her chances <5% if she was playing against someone equally smart".
I've thought about this analogy for less than five minutes, so it's far from perfect. Point is, ASI being made of silicon isn't very central to the problem. There are lots of analogies that are falsifiable, and "ASI will kill us all" can be decomposed into a simple and bulletproof composition of falsifiable beliefs, even if it is too specific to be itself directly falsifiable.
There's are many people with high p(human extinction) whose salary does not depend on their continuing to profess a high p(extinction) -- me for example. Also Yud's friends have promised to pay him what he is earning now if the organization he works for ever has to reduce his salary because the organization lacks funding. They did this to neutralize the financial incentive he would otherwise have to hesitate to announce that his p(extinction) had dropped.
you say "falsifiable" as if it holds some important key, but the concept is really not applicable to predictions that by their very nature, can only ever have one data point
(doesn't mean that it's not a religion or it is, but rather I mean it being "falsifiable" or not, doesn't carry any information to that regard)
The rational methods used to establish a large P(doom).
To definitively falsify the claim, humanity would need to live alongside superintelligence forever without it wiping us out. That doesn't mean we can't be pretty darn sure though, in the same way that we can be pretty darn sure that nuclear war could end civilization.
I wish you'd used "very large asteroid strike on Earth" rather than "nuclear war" because although nuclear war will probably make thing "uncivilized" for a while, I'm pretty sure human civilization bounces right back after even the largest realistic nuclear war.
I wouldn't be so sure about that. 95% of the population will starve to death, global supply chains will collapse, and the world will regress to a pre-industrial-revolution lifestyle - with the critical difference that all shallow oil and coal deposits have been mined out, not to mention many other resources. You're going to have a very tough time bootstrapping it all again. All vital knowledge for running a modern civilization will be lost - most of it isn't even written down now, but is instead implicit in distributed social structures which will all be destroyed. They won't re-form quickly either - travel will be even more risky than in the Dark Ages thanks to fallout. Even a healthy society can coast in a mostly steady state for thousands of years without technological takeoff, and we've had several civilizations in the past who seemed to hold most of the pieces - empiricism, healthy economy transcending food production, precision metalworking - and yet never seemed to quite get there. This won't even be that - it's a restart in a world depleted of natural resources and blighted by death, radiation, mass extinction, and climate change.
"95% of the population will starve to death" is very unlikely although I can't completely rule it out as a possibility. The research on nuclear winter is of a poor quality, and we should be funding better research.
"The world will regress to a pre-industrial-revolution lifestyle" seems to be a retreat from "civilization will end" (not that I agree with it).
>travel will be even more risky than in the Dark Ages thanks to fallout
The radioactive contamination caused by Chernobyl has persisted for decades. In contrast, the radionuclides created by nuclear explosions are of a very different nature and persist for weeks. 3 weeks after the end of the explosions, there will be no place except ground-zero craters (i.e., very small areas) where the levels of radioactivity remain at deadly levels. (But I definitely agree with you that during those critical 3 weeks, almost all people who decide to travel will end up dying unless they know a lot about nuclear war and have good radiation detectors.)
To repeat my overall point: the worst-case scenario for the outcome of AI research should be compared to an asteroid strike by an extremely large asteroid, not to nuclear war—because nuclear war is too tame in its effects to make a good comparison.
The reason I got involved in this discussion is that some people use the possibility of nuclear war as one reason for us to go forward with AI research as fast as possible. Such people argue that a sufficiently mature AI technology would eliminate the possibility of nuclear war. I think that is a fallacious argument become AI "progress" is the more potent of the two dangers.
Those "rational methods" are essentially just pascal's wager. You stick some ~infinitely bad outcome (nasal daemons torturing simulations of you and everyone you love for all time) and/or ~infinitely good outcome (perfect paradise for as many sentient minds as could fit in the physical universe), then you can get whatever conclusion you want by multiplying any finite probability by these things, and even give yourself bonus asspats for being conservative by adding several order of magnitude fudge factors against your claims.
The rationalist doom and sex cult uses convoluted mathematics inspired language to justify their faith-- but at the end of the day it's just faith no matter how they dress it up.
No, it's not Pascal's wager because Pascal's wager works even when the probabilities are tiny whereas Eliezer's p(doom) is about 99% unless AI "progress" is shut down and kept shut long enough that the current crop of AI researchers is going to have to find something else to do for a career.
In other words, it's not Pascal's wager if the very bad outcome is what you naturally expect to happen.
And what I just wrote applies to not just Eliezer and his allies: I don't know a single person or a single published argument that has ever used Pascal's-wager-like argumentation to argue that AI "progress" needs to be stopped. Eliezer is the author of a blog post titled "Pascal's mugging", but that blog post and any follow-up discussion are completely silent on the dangers of AI.
Because they depend on a supply of anxiety fueled doomers to both fund their lifestyle and provide a supply of willing participants for their control and fetish games.
> their lifestyle now depends on "p(doom) is big" being true
How?
Suppose that for the following 20 or even 100 years nothing happens. The AIs miraculously stay as they are today, the further progress turns out to be impossible, there will be no more agents hacking various websites, no cyber warfare involving AI agents, nothing. There will be the year 2026, forever.
How would any of that impact the lifestyle of anyone in the rationalist community? Please be more specific with your accusation.
(Seems like the rationalists are simultaneously the dark lords of TESCREAL, pulling the strings of the world's most powerful billionaires and politicians, but also irrelevant weird nerds who can be ruined at any moment by a smart blogger explaining how their opinions are silly, or even that they know each other.)
The author didn't get invited to any parties in high school, and now they're concern-mongering Yud and Buds over some nerd bacchanals. That doesn't scale well.
The big question is, who's competing to sell us our salvation this time?
I tried too, but found no sentence that makes any sense. It's all gibberish from top to bottom. I'm not a native speaker but my only reaction was "wtf this is all about and wtf are you even trying to say???"
it's the Ed Zitron style of arguing that something big/expensive/unprecedented/hard is bad, going to fail, because [____________].
and if you put something trivially true into the blank the implication seems logical, and the argument feels powerful. but simply putting a true claim at the end doesn't mean that everything before follows from this claim. (and even if they do the argument is not a total description of all the causal chains that influence the outcomes.)
of course it's hard to do a careful survey of the facts and weigh them and then deduce some likelihoods for possibilities. because some aspects are unprecedented (investment ferocity, AI results/progress) while others are very much precedented (collapse of investment bubbles, doomsayers' wrong predictions).
Just because something was called bad, predicted to fail and actually didn't fail, doesn't make it "definitively not bad, because it didn't fail, I told you so".
Case in point - Bitfinex was and is absolutely a criminal enterprise and scam, and it didn't have any backing for years in row. But it didn't fail, because many different criminals in the world had decided that it is quite convenient tool and put a monetary IV line in it. And now ruzzians are buying kamikaze drones with it, selling oil to India with it, drug cartels are shuffling around their money with it and so on. Does it mean it wasn't and isn't bad? Of course no. They simply found a rich sponsor to bail them out.
Or the whole 2008 crisis is another popular example.
I'm not saying absence of evidence of wrongdoing is evidence of absence of wrongdoing.
But note, you have very concrete things to claim about Bitfinex. Criminal enterprise, scam, organized crime funding, facilitating arms dealing and embargo avoidance, and so on.
Then you throw in 2008. It's definitely popular but example of what? Bad thing that didn't fail? (Things did fail, hence the crisis? Or you mean the too big to fail attitude?)
Policy proposals (eg. "coordinated AI safety R&D should be exempt from antitrust regulations" or "TARP made money on paper, but the financial sector should pay for the total cost of the crisis") cannot be divorced from the provenance of the source (sex cult, goldbugs, anticapitalists), but the same is true of bluesky screeds.
2008 was a shortened mention of the event when multiple companies by all rights went "bad", i.e. should have went bankrupt on their own merit and performance. But an external entity bailed them out and so they continue to exist and work later on. But does this mean that those companies (like Fannie Mae et. al.) weren't "bad" in 2008? No, it doesn't.
Same logic with pure AI corpos and adjacent dependents. If the doomsayers would ever be right about them being financially unsustainable, then most likely these companies would never get even close to the bankruptcy, simply because they would be bailed by external entities. So we would never be sure if OpenAI and Anthropic were "bad" or not, financially. Because either critics are wrong, or any corporate deficits would be covered by budget money (maybe foreign budget, it happens too).
Read to the end... The front matter is context to the conclusion:
> This all most recently came together over the weekend, and they got Dario Amodei and Sam Altman and Elon Musk and Demis Hassabis all onto the same page to say they wanted to WAIVE ANTI-TRUST LAWS so that they could "pace", or slow down, research.
> I feel about this roughly the same way I would feel about all the pharmaceutical companies getting together to say they wanted to waive anti-trust laws so they could form a cartel to ban medical research:
> these people are the enemies of humanity.
> Also: Much of AI research IS medical research. They are very literally proposing to stop a pretty good chunk of medical research in its tracks so that they don't have to fear the paperclipper.
> There are many regulations I would be in favor of, but they only ever want to form a cartel or a world government against the paperclipper.
> No other industry has ever framed any regulation in terms of "stop", "pause", or "slow down". Those are insane ways to consider any form of policy. Policy should prevent bad outcomes and encourage good ones, it is never about some specific speed.
> This industry frames its regulations in this way because it has, at its core, a millenarian death cult that worships and fears AI as the cause of the end of the world, that feels compelled to create AI, and that at least somewhat wants to destroy AI as a field.
There are better ways to make this criticism. The main problem I see with this is that it's basically a long ad hominem, followed by some valid concerns about market concentration at the end.
Remove all the ad hominem and what remains is two paragraphs long. Valid ones, though.
That was a figure of speech. It boils down to the fact that, by asking for laws mandating AI research to slow down, the incumbent Big AI firms get regulatory capture and cartel power, which might be the real motivation underlying the apocalyptic language (follow the money and all that).
The other thing that annoys me is that there are plenty of folks in the AI safety space who do not think very highly (at least now) of Yudkowsky. One of the comments I saw recently on a Substack article talked about how Yudkowsky has "beclowned" himself by not updating some of his earlier theories given how LLMs function.
So it's pretty annoying that this article is trying to paint everyone who cares about AI safety is in a "sex cult". There are lots of folks who are genuinely concerned about risks. If anything I discounted a lot of the AI safety folks because I thought they were not seeing the underlying limitations in LLMs. That may still be the case, but agents keep breaking through what I thought were limits on their behavior.
If anything, I don't understand at all the folks who think ASI is on the horizon (< 10 years) but think everything will be hunky dory and easy to control once we actually do build superintelligence.
the sex cult is the sensationalist draw. you don't get readers paying attention in our current content economy if you don't have something salacious
which is also to say - I found it very funny and enjoyable to read and it illustrates how much of dorky little echo chamber that group is. this whole idea that you can become some Spock-like, logic-driven creature in spite of our biological wiring fundamentally opposing this seems to underlay all of this. and the fact that they're all orbiting around one 'cool poly girl' and 'visionary dude' is hilariously similar to the kinds of very formerly-WASPy-and-now-just-WASy hetero polycules that I, in my more youthful and adventurous days, hung out with (and quickly departed after getting icky vibes). like down to the fashion choice and need to constantly prove how smart they each were at all times (do they, too, get really high and put on some anime with way too much fan-service and proceed to have orgies? one wonders)
those are the types of people who always end up actually having the worst coping skills because they can't even admit that they might be emotionally-driven sometimes, or that they have emotions at all that they might need to develop the cognitive skills to deal with in a pro-social way. or they get called out enough that they finally read a little about therapy and just end up weaponizing psycho-therapeutic language for their own ends (instead of, say, practicing the skills, doing the workbooks, developing insight)
which is how your anxieties might lead you to believe that the sky could fall at any moment (and thus the whole nuclear war and the 'don't trust bust us!!' bit)
I think the part that is annoying me the most is that one of these dorky little polycules has somehow found their way to the levers of power and are just mashing all the buttons and dialing all the knobs. these are honestly the last people on Earth that I'd trust with having any kind of influence
> the sex cult is the sensationalist draw. you don't get readers paying attention in our current content economy if you don't have something salacious
Nerds used to be shamed for not having enough sex, now they are being shamed for having too much sex. When will they finally learn to have the average amount of sex, so that the public discourse can finally switch to something else?
They could try just being normal. There's a middle between being unfuckable and hosting rape gangbangs with 40 people. This article actually does a great job describing how the gross, unfuckable nerds are now the ones pushing all of this poly sex cult stuff.
it's also possible to have healthy, non-hierarchical poly relationships. many queer people do it with triads, quads, etc. most of the hetero poly people I met and were icked out by functioned as extremely hierarchical in practice (but they would love to tell you about how they were relationship anarachists or whatever) with a couple at the center and one of the partners being the one that pursued having a lot of other partners while there was hidden pressure for the other to also engage in the practice so they didn't seem patriarchal or toxic (a cruel irony, in retrospect)
that said, my observations have a sample size of like... four couples maybe? (they're indistinguishable from one another now). but the dynamics I saw were four for four which seemed good enough for me at the time to dip out of that scene
One might also argue this is more mainstream in some cultures, and it's just not talked about. In countries such as Japan or France it's not uncommon to have a mistress (or male equivalent) but also having a primary family. That's hierarchical polyamory with uh, less steps.
certainly less calendaring and meeting invites, I'm sure. I did meet a couple of people who identified as non-monogamous well after I had met my very monogamous partner (whom I love deeply and am happily monogamous for) who seemed very cool and were equally put off by the polyamorists they had interacted with. but they were truly diamonds in the rough, at least in my major metropolitan city, and not a single one of them was hetero or close to being a WASP so you're probably right about the cultural tidbit
There's a great movie or Streaming series on this post somewhere.
Sometimes people can be too smart for their own good. Like a frontier model in an eternal hallucination loop, you can create entire seemingly consistent realities in your head and end up bringing others into it.
Well most likely consensual noncon given the amount of vetting present in that chart. It's surprisingly common in women, or just a desire for general "gentle roughness".
And idk why it's really worth mentioning anyway, as a gay furry I suppose vanilla normie straights are STILL hung up on what people do with their sex lives. As long as it's legal and consensual who gaf? Y'all, apparently.
Not opining on this one way or another, but the post buries the lede a bit with a one liner that explains where it's coming from, saying that this is all:
> reasonable evidence that their influence isn't prevalent because they're right, and that they would have ulterior motives for continuing to preach that the world is coming to an end even if evidence showed otherwise.
So how they enjoy their sex lives is dependant on continuing to preach AI catastrophism? I don't buy it. Lots of poly folks out here that probably don't care much about AI one way or another.
No one is claiming that all poly people have some ties to the AI community.
They make a claim about a specific community in San Francisco, that is tied together by a mix of AI doomerism and sexual relations. And it's not at all surprising or some farfetched claim that people who enjoy having sex with someone might adopt their ideas, at least publicly, for fear of losing that sexual connection.
>And it's not at all surprising or some farfetched claim that people who enjoy having sex with someone might adopt their ideas, at least publicly, for fear of losing that sexual connection.
am i naive? in the direction you pose this, i can't imagine it's that common and if that happens i don't feel sorry for our imagined victim. being a bay area guy who attends so many orgies that AI doomerism rubs off on him? and he keeps spreading it to other people and that some substantial bulk of doomerism is due to this mechanism such that it voids the ideological purity of doomerism? i'm not a bay area guy and i haven't attended these orgies but i hope they have other shit to talk about at the orgies than AI
if the relationship is flipped - being genuinely interested in the theory but then being roped into weird sex shit that you didn't ask for, sure that that sounds predatory. a sex cult where AI is instrumental, rather than an AI cult where sex is instrumental. it would be more remarkable imo but i don't know if anyone is saying out loud that's what's happening.
The claim is that these people, in spite of their claims of simply rationally following the evidence wherever it leads, have become trapped in the idea that AI doom is imminent and the only serious concern that anyone in the AI community (at least!) should have. This irrational attachment is then re-enforced not through rational discourse, but through the strong emotional bonds and biases induced by a network of sexual relations.
And it also claim that this phenomenon is happening in a specific group in Berkley, California, and that a large majority of people working in AI safety at the biggest labs are members of this group.
Hate to break it to you, but I don't think the AI doomers have a monopoly on orgies. You name the political / belief system and I'll find an orgy for you by 10pm.
Heh, if you think this post buries the lede, have you ever read anything by the community in question? They seem incapable of saying anything without five part Roman-numbered novel length blog posts.
I think this article kind of misses a crucial point. They weren't always AI doomers, that's a thing they picked up along the way. Their core beliefs are 1. That they are the smartest, and 2. That this makes them morally superior because consequentialism (This sort of follows. If you believe in consequentialism, leaving all your decisions to the superior consequence predictor is surely the moral thing to do)
Maybe one day they will stop believing in AI doom, but they will never, ever stop believing they belong to the intellectually and thus morally elect. The day any of them do that they have either broken with the cult or died.
Well, I used to think like that too. Then I became slightly more spiritual, and the very idea of this sort of behavior is bothering, upsetting, irritating. But it mostly evokes pity. Like becoming aware of someone hurting him/herself in some way.
The way I see it, sex is a sort of portal to another world. This sounds a bit weird, but it really does and should have a great deal of meaning behind it, as it is/was central to our species. If it becomes mundane for someone, this really does ruin a lot of that person's life. Even if you completely remove the judgment element. If sex doesn't occur in the context of a love relationship, it's just uncanny territory. It comes with some emptiness.
There's a proper and healthy way to live. This is from my life experience. Stray too much off-course and you will experience consequences. It's just the way things are.
Amazing how people come with all these rationalizations instead of just accepting that they are not the kind of people who like casual sex. There's nothing wrong or unhealthy about casual sex, it's just not for you. You can have your loving sex all your life, nobody will force you to have casual sex.
About orgies, it bothers me how much people who never frequented these circles have so many opinions about how bad or unhealthy they are. I know religious people who have the "healthy sex life" and are completely deranged. I could say that they are that way because of their sex life, but I would be as wrong as you are.
Some of us have spent countless nights with other kids locked in the basement during parents' swinger parties and/or been there and done it ourselves as adults and found it to be as much an empty experience as the parent poster suggests. Molestation victims also frequently report the same sentiment. Removing love from the equation turns the act of sex into mechanical fluid exchange, and the participants into human livestock.
Promiscuous people are as annoying as those who refuse to eat at the same restaurant twice. You have an insatiable appetite for novelty that you need to see normalized in others-- to defer ever having to face the very emptiness we're describing.
If you actually found it fulfilling, you wouldn't need to sell others on something that sells itself.
Thank you for sharing your personal experience. Not only is what you said compelling, but it again highlights the importance of experience as a path to knowledge. Rationality is brittle and cannot access much of the vastness of human experience.
You had a shitty childhood, I'm sorry. About the rest, I guarantee you that it's just not for you. No problem in that, but don't be mad that other people like the lifestyle. There's nothing unhealthy in that. You can be as puritan as you want, nothing unhealthy as well!
what is so "incredibly" and "imposing" in the parent comment, that saying «my experience is…», «i feel more in it than many imagine/expect», «factually ruins many lifes», and «it just seem working this way», even «remove the judgment element» ??
it is as gently and friendly of a warning as it could be.
i would say rather less nicely: people! don't be degenerate! it does not worth it. it causes harm to you, to neighbours, to bystanders too, even if you dont intend.
Well OP claiming they get "bothered, upset and irritated" by people having "rape kinks" and "pitying them" sounds kind of judgemental to me. Same goes with "don't be degenerate" for sexual behaviours you do not approve that, somehow, also harm one's neighbours for some reason.
Imo judgemental as in starting from "my life experience" and getting to that's "how to live" is a reason to dismiss it as an argument, as others may have different life experiences and yours is not universal.
I'm sure OP would argue that the breakdown of the family structure and social cohesion has a deleterious effect on society as a whole, which includes your neighbors.
Yes and indulging this appetite is much like any other, a good comparison being the other fundamental one: hunger. I think that also frames it in a way that makes it more concrete as to what the negative aspects are as well as showing that total abstinence is not necessarily an unalloyed virtue. If you don't eat at all you will starve but you can easily eat too much or eat the wrong things and it becomes gluttony at the extremes. Coming up with systems that allow you to cheat the negative aspects also turns pathological in the long run. Deadly sins exist even without the religious framing but the spiritual, social and psychological aspects are very real regardless of what anyone professes to believe.
You've made a hypothesis, provided no basis for it, blew past that and built some more conclusions on top of it. Along the way you introduced a few more unsubstantiated thoughts.
The post in it's entirety reads as a religious rant, but it might have had something to offer to the reader if you had bothered to provide some sort of philosophical foundation for why you feel that the middle way is the superior way. And that all excess is inherently not only nearly identical in terms of effect, but by definition harmful.
It's not about their sex life. It's about how sex connects a community of barely legible AI doomerism.
If you don't understand human social interactions very well, or otherwise assume you live in a "world of ideas", then this may not seem salient. However, it is.
Because what we have to explain is why there's a community of people in northern california who deliver unargued prophecies about the end of human civilisation as-if their prophetic ability is a given.
The answer, unsuprisingly, is sex. This is not an unusual answer in the world of "prophetic communication". What, in the end, is prophecy as a social practice? It is some ideological Leader with followers who impart to that leader a 'charismatic power' of foreknowledge based on their attachement to that leader.
What is the mechanism of this attachemnt? In otherwords: how is this style of prophecy spreading around the AI doomer community, if not by plain argument and evidence? Sex.
There are many other... let's call them "orgy circles"... in the Bay Area attended by people who have no idea who Eliezer Yudkowsky is. The majority of AI doomers do not attend orgies. You're reading too much into a single correlation.
I feel like people's problem is less "adults having consensual relations ew" and more "our incentive system is so fucked up that these people are now the intellectual leaders of the AI movement. Not by merit, but by developing a weird erotic exclusionary synchronicity that the rich folks like." Same thing is happening in government too.
I don't want to hear about it: yours, hers (Aella) and I'm certainly not going to tell you what my deal is, either.
Problem is, Aella has made it her business to speak about it publicly, and the parent commenter(and I)'d be happier if we didn't read about it. It's fine to say that.
I'm not hung up about anything others are doing: but I genuinely, passionately, don't want to be informed about it.
That's the solution the parent commenter had! And the gay furry (self identified) complained about! If you have a problem with his announcing the solution (to stop reading) then maybe you should just not read his comment, or my comment, or...
I guess what OP wanted to bring up (as I understood) is that if somebody says "I am straight" it does not usually induce people turning the internet off. But in any case they are free to choose that and that does not really matter in anything (a person not wanting to read sth). I still think it is worth bringing it up as OP did.
I don't think the woman in question has participated in this thread. On the other hand, all the pleas to "stop making me read about sex" reminds me of:
"Why Do All These Homosexuals Keep Sucking My Cock?"
> Just don't read her stuff if you're not interested.
This retort is disingenuous when almost everyone is on algorithmic social media and trying to chase clout and drive revenue with engagement bait.
I would never read Aella's Substack in my free time, but I have a good idea who she is and what she posts just by virtue of having a feed that mostly gives me tech and politics stuff.
I have no idea how trustworthy this study is - but it claims that 62% of women had such a fantasy at least once. How is that not in line with what the OP said about it being "surprisingly common"?
Many rock climbers have felt the "call of the void" when climbing, I've felt it myself on a multi-pitch once, I'd hardly claim they are suicidal or have suicidal ideations and would not identify myself as either. I think there needs to be a certain level of intention to make the fantasy reality for it to be a kink.
I suspect the misunderstanding might be around the precise meaning of the word "kink" then.
I think in one common understanding of the word, that I assume the OP was using, having a taboo fantasy that you merely roleplay with a partner you are deeply intimate with, in full consent from both, can very well be classified as "a kink", even if what you're experiencing is the not really SA.
Based on your response, I'm assuming that in your understanding, it would only be "a kink" if it was a desire to actually experience a real sexual assault, which of course no one, or perhaps only an extreme extreme minority, actually wishes for. I don't believe OP meant to imply that this is at all common.
> I think there needs to be a certain level of intention to make the fantasy reality for it to be a kink.
Hopefully this isn't to imply that people with fantasies actually want to have non-consensual sex in some capacity. If so then this completely misunderstands sexual fantasy!
I don't really get why people go on about Yudkowsky in this regard. I mean serious thinking about AI goes back at least to Turing in 1950 with his test and prediction computers would get there in about 50 years and presumably keep improving. And people have worried about it since then.
Like in 1949 Norbert Wiener saying of AI type stuff "every degree of independence we give the machine is a degree of possible defiance of our wishes."
Then 30 years later this Yudkowsky is born, later is considered a bit of a crank, goes on a bit about the same old stuff when he grows up and everyone makes out like he invented it?
I've got a lot of respect for Turing, Wiener and Von Neumann but haven't bothered reading Yudkowsky. Is there any point?
It seems a bit unfair to the pioneers of the field to go on about some Harry Potter sex cult guy who repeated their ideas decades later?
Regardless of anything I would have more respect for anyone who didn't explicitly optimize the atomic bombs on Japan for civilian casualties like Neumann did.
Yudkowski is being derided because he wrote some weird fanfic which may or may not be problematic (I didn't read it, I dislike HP too much). And apparently "is considered a bit of a crank".
Neumann actually fucking did it, his calculations for maximum civilian casualties make him quite directly responsible.
I don't think you need to compare Turing, Wiener or Yudkowsky to that mass murderer.
In case the author is reading this: I’m getting a Cloudflare worker exception when accessing the page using one of my uplinks, possibly because of IPv6.
TLDR: awareness of the SV "rationalist" polycule has breached containment and normies are reading about it now. [0]
Maybe a "better" source is the NY Post article [1]. The information itself isn't really news - it's even largely available on wikipedia [2] [3] - but the fact that it's hitting mainstream-adjacent news outlets is novel.
> Yudkowsky's most viral idea, the one that infects the most people and spreads the farthest, is that AI, if we continue to develop it, will inevitably 1) achieve superintelligence ("RSI"), 2) escape human control 2) kill everyone ("paperclip them", "doom")
Any actually-reasonable person who was around for the founding craze of this era, ie., New Atheism, quickly realised that there was a deep pathology at the heart of many of its zealots, perhaps summed up by a desired to "recreate religion in their own image". I've walked into many a room of people screaming that teaching children falsehoods is literally child abuse. Any one who knows much of Richard Dawkins, Krauss, et al.'s background with women shouldnt be surprised either.
It's californication into the "rationalist" community as it currently stands is no surprise. Every person I've met deep in this community either seemed to me a narcissist (if male) or, politely and mildly, coquettish (if female). With several relationships I've observed a clear union of the two. I hadn't yet put this so clearly together with the "AI Saftey" crowd, despite being directly adjacent to it all for quite awhile and seeing the cross-proliferation.
My own analysis had concerned their philosophical naivety, general lack of actual computer science knowledge and generic shallowness of understanding of any domain outside their own research and a kind of "science-fiction" version of philosophy and rationalism. I hadn't connected this yet to their strange personalities, but the apoclypticism fits and neatly bridges these together.
So what became of the New Atheist project to create a "secular religion" -- an apocalyptic sex cult furnished by a blend of sci-fi, pop-philosophy, and cute parables from polymath medical professionals. All in all then, it seems they succeeded.
> It's californication into the "rationalist" community as it currently stands is no surprise.
This is the minority split. Most of the online new atheism writers became progressives/social justice/"woke 1.0" people. That trend has receded, so I couldn't tell you what they're doing now. (Probably blocking each other on Mastodon.)
The rationalists are the ones who stayed behind and decided to stick to doing logic and math in their heads instead of thinking thoughts the normal way.
That has not been my impression. Who is the most prominent example? Some of the most prominent figures I know of were Richard Dawkins, Christopher Hitchens, and Sam Harris - all prominent right-wing figures who were rewarded for using atheism to attack Islam in particular during the Iraq War.
None of those people are remotely right wing. Christopher Hitchens was outright Marxist. All believe(d) in liberal, progressive values, which is precisely where their objections to theocracy came from. If you terminate your thoughts at "religion bad = intolerance = right wing" then I am afraid you will be led to severely misguided conclusions.
I don't believe any of these people are relevant or prominent, but the unimportant people I was thinking of were bloggers like PZ Myers, I think they called themselves "Atheism+" or something.
> Richard Dawkins, Christopher Hitchens, and Sam Harris - all prominent right-wing figures
Are they? That's um, a scientist whose pop book on genetics I read once, a guy I read in The Nation once, and a guy whose book on Buddhism I read once. I remember the first two evolving into GenX curmudgeons (probably the lead poisoning) and have never thought about the last one again. Still, those aren't the right-wingers I'm familiar with.
I wish there were more discussion on HN about the various plans that the various labs have to keep AI aligned or maybe about the various arguments for why alignment will be easy or happen naturally, so there's no need to carefully plan for it.
But what HN has instead is extensive discussion about whether people who oppose the continuation of AI research are morally degenerate in one way or another (or weird or ridiculous)—in other words, attempts to cancel them (or laugh them out of the room) before the listener hears their arguments.
Also, if you're going to talk about the corrupting or biasing effects of a flow of money (like the OP does) you should at least mention the absolutely huge flow of money going (from investors and users) to the people building AI—or or the $140 million that has been donated to the "Leading the Future" Network (a pro-AI political action committee) or the $3.5 million OpenAI has spent on federal lobbying or the $3.1 million Anthropic has spent.
I was invited to a few orgies with them; I politely declined, they were very respectful of my decision and I wasn’t asked again (which in retrospect must have been partly due to deliberate coordination - fairly advanced social technology, imo). This didn’t affect my standing or acceptance with them. It may well be a sex cult, but if it is, my experience is that it so thoroughly behaves like a regular community in so many aspects that it at least calls into question the predictive usefulness, if not the accuracy, of the label “sex cult”.
It sort of seems like the community is criticized for truly believing and living what they preach... They're the most sincere - if you thought the world was ending, you'd maybe do your best to stop it while enjoying the last bits left, while spending time with like-minded people. They're being criticized for not being hypocrites? I mean sure seems like there's lots of conflict of interest, but isn't there always? What humans have ever been objective in anything they do
The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?
The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.
You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.
You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.
i read AI 2027 at the time and thought it was nonsensical. however i was not aware of the conflict of interest running through the philosophy itself, the safety institutes "independently" monitoring ai, and the former public benefit orgs like openai and anthropic creating ai (now de facto fairly evil for profit companies).
it explains so much now.
it would explain why redwood and METR were chosen and apparently willing to immediately write a whitewash report for openai blaming the agents.
it explains why anthropic is infantilising its users, anthropomorphising its product when there is no scientific consensus of llm consciousness, or accepting of outlandish and unreal claims about human extinction when they are lacking in evidence.
who knows, it may also explain and fund the recent media blitz by jacob coxon, which has led to the recent virality of ai safety.
sure, yeah, I've seen how the sausage is made on some of this stuff, and its not pretty.
However this only claims that they are not good at creating effective and trustworthy orgs, it does not address their claim that it is not possible to crate powerful, aligned, and controllable AI. Both things be true.
for what it is worth, i find current frontier ai impossibly stupid. you ask what is the date, it doesn't know. if you set a system prompt to repsond in 5 sentences after 100k context, it can't, the context rot just doesn't permit it. tool calling is not intelligence, it doesn't fix the underlying issue.
it is uncontrolled in the sense that a rock is uncontrolled if you throw it at someone's head.
it isn't good enough for anything important. you would not trust it to run an alarm in a power station.
of course it is not general ai. ask an llm to drive a car. you need to see in fractions of a second ideally. the llm might brake after 20 seconds. i think llms are doomed here. we could come up with a different architecture but then we would be back in the land of fantasy.
we are stupid but hopefully not that stupid. we are dealing with something computer based, luckily we can survive without, with some thinking about it. i would start there instead of with the AI part.
I mean, this is a rather fun article to read but I'm not entirely sure about the thesis.
> AI Safety Is Mostly A Sex Cult.
> I don't think these people should make policy. (1/?)
One of the learnings from things like the whole Epstein saga has been there are a lot of sex cults in the policy making world and, yes, the people making policy generally shouldn't be. But I'm not sure this article actually comes up with a good explanation of why these particular people shouldn't be making policy. People having active sex lives isn't a disqualifying trait. Being in a cult doesn't disqualify people from being in politics. It is routine and expected that fields will have a lot of similar people in them (as I recall the overlap between security researchers and the furry community is unusual too, and that is hardly more important than justifying a passing comment).
It is more routine than the article writer seems to think for all the people in a small field to know each other and get on really well.
