I think it does gel. You're right that the comment you're replying to is weakened by that fact.
It gels because they solved about 700 problems out of 4000 attempts. And they threw a ton of resources at each. So "per problem" it's less than 1 in 5, "per token" it's probably not a great record. I mean it's astonishing that it's possible at all of course. But on average, even an expert with a top model will probably have an underwhelming response to the vast majority of prompts.
and I'm guessing it was with "tool calling" ie. they can write a bunch of Lean proofs and hit compile and basically brute-force a whole bunch of hypotheses quickly?
Ie. the LLM itself isn't as "smart" as the combination of LLM, Lean, looping, search and vast gpu/cpu-time.
In my experience they write an informal proof first, check it without Lean, and only then turn it into Lean code. That last step sometimes takes them much longer than checking their proof without Lean, which is a problem humans have as well. What they're truly bad at, however, is explaining their proofs. Until they improve at that, we'll have to keep resorting to Lean to check whether they solved something correctly.
I'm less sure about the following, however: I think that while current LLMs are bad at explaining their own proofs, some models are great for discussing older, human-written proofs. For this use case, from best to worst, Gemini > Claude > GPT-Astra >>>>>>>>>>>> GPT-Sol.
I definitely don't know but I got the impression it's the opposite. Based on the observations about Navier Stokes and the natural language proof not matching up with the lean proof and other things. They generate lean and convert it into natural language at the end. But really I could be wrong just giving anyone interested a lead to check out.
I can believe this is true. It's great at going out there, connecting ransom dots in very clever ways, sometimes stumbling on something actually funny.
I'm curious, did you test with the 200 jokes if it could guess which were the beat ones according to a human audience?
In images, music, games, it usually can't. So the process of validation is offloaded to the audience. I'm very interested in seeing where this takes different platforms. I bet Spotify can pull off such a validation process. Music is passive, tolerating a bad song doesn't require much from you. And they have a massive audience. They can try out AI generated mess with a very low success percentage without saturating their playlists and alienating their users. But can Steam tolerate it? Trying out a game is a much bigger effort than listening to a song picked for you. Customers are already reacting to the number of demos in Steam Next Fest. The saturation point is much lower.
And what about jokes? What happens to r/jokes if even 10 standup comedians are validating 200 jokes per day?
And we see it in software too. QA is already being offloaded to the customer.
I want to see where this goes, even if I don't like it.
But if you find a media where AI can also validate the output instead of having to offload it, that's going to be very valuable!
I wonder how much different that is from humans, even professional comedians. You don't usually get to see all the jokes that comedians try that don't work, unless you're an avid attendee at comedy clubs.
For comedians, perhaps. But there are plenty of humans who aren’t very good at comedy.
In these LLM comparisons, people tend to compare them to the best humans, not average or below average. It’s a kind of motivated reasoning, with the goal of deprecating the abilities of LLMs.
Well, you did ask for a comparison to professionals.
But I think humor is pretty easy to rate for a human. Did it make you laugh? Yes? Then it might be a good one. This simple human filter radically improves your efficiency as a joke validator, leaving any LLM far behind.
For pictures I'm not sure if humans are much better. For music, yes, after a few listens (recognizing a hit usually takes time). For games, definitely yes.
They can still unintentionally be funny sometimes though, especially certain quirky models. Mimo 2.6 Flash is my favorite in this regard because it will become absolutely unhinged in ecstasy when proposing an idea before even trying to verify if it's correct or see if there's a better one. Probably my favorite sequence from an LLM ever was watching Mimo try to diagnose a bug and coming up with a series of theories that it introduced as "NEW IDEA!!!", "ANOTHER IDEA!!!", "BETTER IDEA!!!", "EVEN BETTER IDEA!!!", "BEST IDEA!!!", and then "ANOTHER BEST IDEA!!!". It's been months and when someone is listing ideas I'll think of the phrase "another best idea" and need to stop myself from laughing.
I have to confess to being wrong in exactly that way in the past. On the other hand it is probably always good to behave as if the next step up in the hierarchy is still available. There's no point letting these things spoil your day twice.
Now I'm forced to think about it much more carefully. I don't believe that current LLMs can find every possible proof on their own from now on. I also agree that the PR returns have fully diminished. Given how hard it's been in recent years to fund e.g. a poorly paid postdoc, I doubt there's a ton of money lying around for getting the best models to work on mathematics problems. I think there's a chance we're heading into a really good era. I was able to make so much progress back when Google search was a useful research tool...
Once you decide that you believe what you believe because you are very smart and almost always believe true things, and the things you believe just happen to be the true ones (I am being unfair but I think this is the psychological effect of this so-called rationalism) it's not a big leap to: everything I have to say is worth hearing.
> So why, then, does truth sometimes engender hatred? Why does the servant of God come to be an enemy of those who want a happy life, even though true happiness is joy in the truth? The reason must be this: Our love of truth is such that when we love something that’s not the truth, we pretend to ourselves that we we love is the truth. Then, because we hate to be proved wrong, we’re unwilling to be convinced that we’ve deceived ourselves. In this way, then, people hate the truth for the sake of whatever it is that they love more than the truth. They love truth when it shines warmly on them, and hate it when it rebukes them.
It’s a feeling that I got from reading his stuff over the years. Asking the AI just now comes up with a number of examples where he himself admitted to it. I also found him increasingly disingenuous.
Can this truly be the reason? Aren't there infinitly many non-verbose writers out there who believe themselves to be correct just as much?
I suspect a reason for the length of rationalist writers is often rather that they value being objectively correct more than others. Length is just an attempt to be less wrong by including many aspects of a topic to paint as complete a picture as possible.
Additionally people often write what they like to read. It may be shocking to many, but some people may enjoy reading long form text that includes a lot of information that allows them to form their own thoughts. Just like some people enjoy noise music others would characterize as a frontal assault on human hearing. Those people would often find 90% of the music regular people enjoy mind-numbingly boring. Human taste is surprisingly diverse.
That doesn't mean that there isn't objectively bad rationalist writing (or objectively bad noise music). E.g. sometimes the train of thought is entirely missing and instead of building an argument it feels more like a lose pile of material that the author stumbled upon by chance in that specific order. If it is declared as that in the start, fine. If it pretends to be a great essay on the nature of $topic and then derails, that means the promise isn't kept.
I always get the impression that they correlate length and correctness. As if the Perfect Best Answer to some question is an infinitely long essay that resembles someone’s rambling mind obsessing over a particular topic, forever.
Which is part of the reason why many people find their actual arguments lacking, and the attitude self-absorbed. Because they don’t seem super interested in arriving at a precise argument, but rather in demonstrating the operation of their mind.
Length is no substitute for substance. However, reality, as they say, has an surprising amount of detail. That means if someones goal is to adequately represent it you will not be able to do so in a engaging, polemic one-liner that puts everything neatly into a box.
Note that within my argument the arrow of causality points in one direction: To explain complex topics adequately a certain amount of length is needed. That obviously doesn't mean long texts are automatically the more adequate explaination of a complex topic in all cases.
The linked post points out that writing from the rationality community often spends excess time on insubstantial parts of an argument, which can actually distract focus from the truly essential reasoning and doesn't help the reader. I've read a lot from LessWrong/rationality adjacent community and agree with that observation.
Don't know if it was intended but there's something to be said about the discussions happening at length on www.lesswrong.com and I felt I had to acknowledge your penmanship.
On the actual topic I for one think some people like reading themselves just as much as hearing themselves thus they're writing for their own audience of one.
Yeah sure. Writing is a way of structuring your thinking. Now whether you need to air that out to the public is a question of character. I tend to do 90% of writing purely for myself and need to force myself to write a blog post, for others that may differ.
Some of my favourite websites are weird 90s website where some weirdo decided to dedicate a website to their extremely obscure niche interest in way more detail than anybody would ever need.
Oh God, that's why pastors keep giving long-winded sermons that are claim after claim after claim and without any argumentative structure (and divine inspiration stands in for self-belief in one's own intelligence)
Such a great description of Rationalism: "I am very smart and I made up a methodology that is infallible therefore..."
It's like a secular IQ-Calvinist take on infallible religious authority.
An ingredient you're missing though is what I call "intellectual edgelording." All Rationalists don't do this, but it's endemic in the community and has helped give birth to things like neoreaction (monarchy is good actually), extreme "if anyone builds it everyone dies" AI doomerism, and has made them very open to things like "race science" and other "forbidden ideas THEY don't want you to believe!"
They'd never characterize it that way though. They launder it through "giving a fair hearing to ideas that are outside the normal Overton window" or similar framings. But if you look closely, the ideas they give that fair hearing to all have a certain kind of bent: edgy, elitist, or validating to their belief that intelligence can be reduced to a single scalar 'g' and this confers some kind of divine authority.
I think the AI doomerism might stem from this. If intelligence can be summarized by one scalar value than if we build something whose 'g' is on average much higher than ours, it will of course annihilate us. The other unexamined prior there is that intelligence is necessarily supremacy which is necessarily dominating or genocidal. Which leads back to why they fall so easily for fascist-adjacent and racist ideas.
The least charitable take I have on that line of thinking is: they at least secretly believe that lower-IQ people should all die, so they project that belief onto their hypothetical superintelligent AI. It would, of course, come to the same conclusion, right?
I think their self-criticism is an act, a sham to make their arguments look stronger. They pick battles they can win and don't engage with ideas that really are challenging to their reasoning methods or priors, especially if they come from outside the rationalist community.
It's like this:
I can write an unbreakable cryptographic system by inventing one, going through the motions of making up attacks and showing that those attacks are impossible, and then concluding my system is unbreakable. Anyone can design cryptography they themselves cannot break.
Ask any cryptographer how well that works.
This is pretty normal human behavior, but it's not rational.
Yeah but would they ever admit they were wrong about things they published themselves? Are they open to criticism they didn't come up with themselves?
I feel (but gut feeling, I'm just an internet commenter / shitposter that doesn't have the time or inclination to fully engage with these people) that the bigger rationalist bloggers all stand on a tall soapbox, broadcast something, then disappear into their own head again.
Correct. They basically just want intelligence to determine social ranking in society.
Whether society eats the people at the bottom or not doesn't matter to them, they just want to be at the top.
If AI is at the top then they will be thrown into the hypothetical meat grinder [0] they thought they were safe from. So AI is the devil to them because it breaks their ideology at its core.
[0] Is it hypothetical if a rationalist is scared of it?
I think a lot of people are tempted to define the people they disagree with as idiots. Not all of them write down their thoughts at length. Both the verbosity and the edgelordery seem to be part of the culture of that community rather than inherent consequences. I think the sequences and HPMOR started or popularised a sort of norm that length is indicative of serious thought.
When I was younger I kinda liked edgy topics and stances for the sake of being edgy. I can easily see how self identified rationalist people that want e.g. racial superiority to be true persuade them self into believing that by looking at economical outcomes and crime statistics handwaiving away historical circumstances and the pressure on the bottom rungs of society which are in the bottom be design.
Their leaning towards fascism is some sort of reductionism for these people I think. In essence political violence make the perpetrators more powerful than if they were nice playing liberals given the same popular support. Thus fuscism. Q.E.D.
Way too little focus on second or third order effects.
Rightwing radicals that 10 to 30 years ago would say they are libertarians are now fascists instead. It is really strange to witness. I think there is a lack of focus on second or third order effects. It is just problem and superficial cause. How hard can it be, just do it, etc.
On the contrary - I found rationalist bloggers to be the most willing to change their mind in face of new evidence out of all groups of people I've ever met o the internet. Although HN commenters are a close second.
I think rationalism might be one of those things where those self identifying and those practicing might only be slightly intersecting.
So it does depend on what you mean by rationalist.
I think it is vastly more likely that those who feel like everything they say is incredibly important might just be idiots who self identify as rationalists, and those who are rationalists are doing more thinking than broadcasting.
> You could say there is little overlap between people who are good at basketball and those who say they are good at basketball.
Right, one could make that statement. It would be silly, but they could make it. I think most people who consider themselves good at basketball are probably better than average at basketball.
> That statement does not have any effect on your ability to judge them by watching them play.
Is the notion that someone who's not good at basketball can't recognize a good basketball player?
I actually think my metaphor is also true. Based upon what I have observed in real life it seems to hold up.
I should note that you changed the statement from say to consider themselves. It is entirely possible (and rather frequent) for people who do not consider themselves good at basketball to say that they are. People who actually are good at basketball quite often do not feel the need to say so, because it is apparent when they play.
I can change "consider themselves to be rationalists" to "say they are good at basketball" because that is just a change of premise. It leaves the structure the same.
This unfortunately doesn’t help if it isn’t paired with withholding judgement in the face of insufficient evidence, but rationalists don’t do that. They just pick a model they like and continue, and the burden of scrutinizing that model is on others.
Making rational decisions means picking the most responsible option given imperfect information. It doesn't mean delaying decisions until you have perfect information because that point never comes.
What's sufficient information is also a huge part of the debate. See: should we start worrying about AI safety or not yet?
Opinions based on insufficient information help immensely in determining whether we have sufficient information, and also raise awareness of important topics that we need more information on. These blog posts serve a very important role in the information exchange ecosystem and the whole world would be worse off if all these people held off until they're 99% certain. Nobody would be talking about climate change, about economy, about quantum gravity, about ancient civilizations, about prehistoric animal species, etc. etc. and so on.
Using the easy to defend claim (acting without perfect information is necessary) to defend a much broader one (my judgement is rational and justified based on the information I have) is a classical motte and bailey fallacy.
People in the rationalist community don’t see it, because ironically they’re not rational enough. But they use the language and essay format to make it seem like they are.
Rationalists seem to believe that their ability to convince themselves of anything given enough evidence is a sign of intelligence.
It's not. It's a sign of being disconnected from yourself and your true values. There is a great amount of wisdom and intelligence in properly honed intuition.
Rational thought should help you understand yourself, not disappear.
What you're really saying is that intelligence is multi-modal, what we'd call an "ensemble" in machine learning terminology. Intuition is another mode, a cognitive process that doesn't look or feel like symbolic reason but has evolved to do something that fills in symbolic reasoning's blind spots or serve some other evolutionary function.
I agree.
I might be wrong, of course, but I would argue that general intelligence -- meaning intelligence reducible to a single scalar 'g' -- does not exist. All intelligence is a hybrid ensemble and an adaptation to a particular niche or set of niches. I'm a reasonably smart human but I'd make a really dumb tree or dolphin.
To get started, look up the no free lunch theorem.
There's room for debate here though. I think our technological journey with AI is going to help answer some of these questions.
With the Rationalists, I get the sense that 'g' is a faith, a prior they won't examine.
Rationalisation is the default mode of operation for humans. Believe first; find evidence to justify it. That's what everyone is doing. Not that self proclaimed rationalists would be free of that. But pretty unfair characterization for a movement whose key goal is to that as little as possible.
The heart of science is not doing this but checking your reason against reality constantly.
It took us thousands upon thousands of years to arrive at this simple idea because it's a blow to our egos and to primate hierarchy and its pretensions of authority. If I am big monkey, I must be right.
Their stated key goal is avoiding it, sure. But in practice?
My idea is that through first principles and made up probabilities, they can reach any conclusion they want. But there's a veil of authority: "I'm a rationalist, therefore I arrived this conclusion, surely not from rationalization, but from first principles, probabilities and logic! It doesn't matter if it sounds insane, because it uses first principles, probabilities and logic!".
Not sure - perhaps true for an oversocialized subset or people who try to wield and gather social influence.
Robinson Crusoe has no use for rationalization. Him rationalizing the berry he ate is not poisonous, or that the shack he built will hold up in the storm is outright dangerous, and definitely useless.
Totally agree, it's like their singular defining characteristic.
It's just that: wanting to have more accurate beliefs. Believing that this is worthwhile even if it's hard to push against human nature of habitually "choosing" beliefs.
Pretty unfair of you to conflate a movement's stated goals with its behavior when the comment that you are replying to is very obviously talking about the latter. So rationalist!
As a theorist, it's sort of true that I "don't care" about galactic constants. I'm interested in the structure of computation for philosophical reasons. But I really do care about the model of computation.
To put it another way, the constants inside a big-O are not really constant: they depend on what you assume is a constant-time operation.
About big-O constant, I came to the realization that in the real world, constant time is actually a log at best, but more likely a cube root.
The simple idea is that the most basic thing, that is reading the number "n" depends on the number of digits, so that's a log.
But in most algorithms, the "n" refers to the size of a data set. This set is usually stored in some kind of memory, and memory is a physical array of little cells in 3D space. Accessing that cell means travelling through the array, with a time to reach it proportional to its distance, and since capacity increases with the cube of the distance, the time it takes to get the data is proportional to the cube root of n.
The conclusion, to me, is that when complexity gets down to less than a cube root, mathematical abstractions start to break down and you need to consider the actual machine, in particular its cache architecture.
No, memory reads do not depend on the address distance, unless you create a list.
Ordinary memory is not a list. It's a flat space read in O(1).
In theory. In practice there are multiple levels of cache, and cached reads will be much, much faster than main memory reads.
So if your entire data structure lives in cache and you repeatedly loop through it, it's going to be processed much more quickly than a bigger and more spread-out structure.
There's also paging, and alloc/dealloc to consider, and maybe garbage collection.
By the time you include all of that big-O estimates can become wildly misleading generalisations, unless your n is always >a million or so.
They mean physical distance, not "address distance".
And in the general case, memory reads do depend on distance. Hence:
> In practice there are multiple levels of cache, and cached reads will be much, much faster than main memory reads.
This is why we have the cache levels in the first place. And the hierarchy continues: L1, L2, RAM, block of spinning rust in the closet, "the cloud" somewhere... There is only so much storage you can fit close to core! More = physically bigger = physically further away.
Sure, within a level might feel like constant time. But that's because the optimal case for that level is slowed down to match the suboptimal cases. The step discontinuities in access time are caused by our inability to engineer a system to manage this tradeoff continuously, not a fundamental law.
Cache is not fast because it's physically close to the core. Well okay, closeness does matter but it doesn't matter as much as the fact that cache is static, i.e. it's (effectively) made of flip-flops.
If you build a computer with DRAM 10 cm away from the CPU and static RAM 50 cm away, access times for the static RAM will still be much lower than access times for the DRAM.
Why don't we just build all RAM as static? Because it's more expensive per bit, it uses more energy, and it requires a great deal more silicon area.
When we are talking about galactic algorithms we have long since left the realm of things that might fit on one computer. Some of these algorithms are only starting to become faster on n so large that we have trouble fitting the data in this solar system. At those scales the distinction between the type of RAM is irrelevant. At that point "how long to read memory address X" is pretty much the same as "how long does light take to reach X" which is pretty much "how far to X" which scales with a cube root. (Well, probably a square root if we look at physical limits for weird reasons)
For any big enough memory the speed of the memory reads will depend on address distance, at least due to the finite propagation speed of the data that is read.
Sequential reads from adjacent cells will be pipelined, so they can have a throughput that does not depend on the distance in address, but the latency to start another sequential read stream from a different address must depend on the address, for very big memories, e.g. of rack size, where the propagation from a corner of the rack to the other corner takes a long time in comparison with the memory clock.
My understanding is that in the limit (i.e. before you create a black hole), information density actually scales with surface area, not volume, so square root. And then of course at that kind of density, you'd have to account for time dilation as well...
"Information density" is a ratio, i.e. the ratio between the amount of information and the length, area or volume in which it is stored, so it does not "scale".
The amount of information at high enough amounts scales with the volume in which it is stored.
Because of the need to be able to reach the cells that store information, the density of information is normally high in 2 directions an low in the 3rd direction.
This is why it is useful to distinguish between the density of information per area, which is determined by things like the size of a magnetic head or the resolution of a photolithographic process, and the density of information per volume, which is determined by the density of information per area and by some kind of layer thickness, which is usually determined by mechanical considerations, e.g. to ensure enough strength for a tape or a disk or an integrated circuit package.
Like I have said, at high enough amounts of information you reach an almost constant information density per volume, like in a rack full with HDDs or in a magnetic tape robotic library.
Yes, misspoken. Anyway, not that I'm an expert here but my understanding is that at maximum density, information in a region does not scale with the volume in which it is stored; it scales with the surface area of the event horizon it creates (at least for a Schwarzschild black hole). Is that incorrect?
The amount of information you are talking about is quite a bit lower than the amount I'm talking about. Big O is about limits, so if we're going to bring physical limitations into play to say there's some limit to density, you'd might as well go all the way. It's not limit as n -> 10^18; it's limit as n->infinity. You can't keep adding information and still get cubic scaling.
Some of the computation models (including the ones in some of the OpenAI papers in question) avoid this particular concern by working in a multi-tape Turing machine model. One might quibble about the energy cost of moving an entire infinite tape (because it’s multi-tape and, for bounded communication code, the heads need to stay near each other), but the machines really are local in a fairly strong sense.
I contemplated the article and it occurred to me that integer matrix multiplication is probably O(n^2), without even any galactic constants. But it will need unit-cost integer math.
First, find a sufficiently large base b so that no intermediate calculation exceeds b-1 (one pass over the matrices should do the trick). Then encode each row of the left matrix as a big value A_i = sum_k a_ik * b^k. Encode each column of the the right matrix as B_j = sum_k b_kj * b^(n-k). Compute A and B explicitly (takes O(n^2) integer operations).
Now you extract the answer one element at a time. A_i * B_j has a bunch of terms including sum_k a_ik * b_kj * b^n. All other terms have different powers of b. So do an integer division by b^(n-1), discard the remainder, then take the result mod b, and that gives (a*b)_ij, costing one integer multiplication and two divmod operations per output element. Problem solved. Can I publish in FOCS?
This is, of course, a wild cheat: even with OpenAI’s latest and greatest galactic integer multiplication algorithm, this would be asymptotically slower than the plain high school matrix multiplication algorithm in any remotely realistic model of a computer.
I wonder whether the OpenAI papers contain cheats quite as outrageous as mine :)
(The moral here is that exact math is quite powerful, and exact math in time sublinear in the number of bits of precision is a great way to steal some performance.)
I haven't come up with anything using natural integer operations. I can come up with really disgusting hacks using highly unnatural integer operations, but that feels too gross to even include in this category of cheats -- if you're willing to use unit-costs unnatural integer operations, then you might as well give up: you can do literally any operation on inputs that can be kludged into a integer in constant time.
Czech republic has some really nice quarries and dams where you can swim for free or next to free. One thing I've noticed is that Czechs only seem to go to such places when it's really really warm. So they're often empty even when they're perfect for swimming. At least the ones near me in Plzeň. I grew up swimming in the Atlantic so my idea of swimming temperature may be different to average Czechs!
I'm hoping and actually expecting that eventually there will be a simple way to locally host a small model that is good enough for simple things. For a lot of use cases, the capability of free chatgpt one year ago was more than enough.
Maybe there already is! But last I checked it was a little bit of a mess of manually installing things from multiple websites with little documentation.
Here's a dumb question. If there was just one tax, the lvt, what would stop businesses sort of rediscovering income tax.
I mean putting a line on your pay slip called "lvt contribution" and then state the pre-lvtc salary in job descriptions and stuff like that.
Similarly with vat and related taxes. The incentive in both cases being to signal: here's what we would pay/charge if it weren't for the government. Which is presumably the current incentive to state before tax salary/cost (this assumption could well be totally wrong).
I think the post is actually saying exactly the opposite.
It is saying that savoring a life fully means savoring all of its varied moments, even the bad ones. Enjoy the highs, but also pay attention to your suffering in the lows because those moments are also meaningful pages in your story.
It gels because they solved about 700 problems out of 4000 attempts. And they threw a ton of resources at each. So "per problem" it's less than 1 in 5, "per token" it's probably not a great record. I mean it's astonishing that it's possible at all of course. But on average, even an expert with a top model will probably have an underwhelming response to the vast majority of prompts.
reply