It isn’t really controversial, of course, but it’s interestingly subtly sycophantic and self-promoting? You can see why RLHF might lead to such answers.
If you ask people “what is a smart sounding answer to this question” you may well arrive at this result more often than when asking for their actual opinion?
Yes, I guess it matters a lot whether this was quite close to the best they could do or the best they could do given specific resource constraints or whether they just didn’t bother to try doing better (eg to invest more tokens into readable papers)?
> I guess it matters a lot whether this was quite close to the best they could do
As the author of the post points out, there is no way this is “the best they could do”. It’s a write up that didn’t involve someone with the math + communication skills required to clearly explain the result.
In this hypothetical case funding agencies would probably happily pay for “alien maths translation grants” and you could write papers about advances in that field and get jobs etc. So arguably for better or worse it would create a specialised academic cottage industry that somehow interfaces with the rest of maths.
Value judgment aside: Probably because it works well empirically.
If you’re a larger scale donor and you’re interested in maximising impact, you may agree to a matching program type setup to crowd in more donors. Also you feel smart because you crowded in more donors and the donors feel like they got a good deal. So several people feel they can attribute the same counterfactual impact to themselves ;)
If you're a charity, you aren't optimizing for max(value_generated) in an immediate way.
If your mission is to help children with cancer, you don't want to spend 100% of your funding on chemotherapy and drug development, because that way, you eventually run out of money (donors die and their cards expire, and you aren't spending on marketing to bring in new ones). You may spend some of it on buying a fancy yacht and organizing a ball for the rich, which will bring in 4x as much in donations as you spend on getting it to happen, and that's the money you spend on chemo and new drugs. If you care about as much money being spend on cancer research as possible, that's potentially a better way to make it happen than spending your donations on cancer research.
Imagine a donor is willing to give you $1m, but you believe you can get up to $500k in extra donations from strangers on the internet if you do donation matching (it's essentially a free marketing campaign). So you ask them for $500k in hard cash and $500k in matched funds. Even if you overestimate and only get $400k in donations, you're still getting an extra $200k of free money.
Yeah, iterating that a bit further is why paying for political lobbying can be worth it compared to direct donations.
However, the less direct your theory of change, the harder to quantify impact and the wider the posterior distribution tends to get.
Some of the most impactful things I’ve donated to (imo) have a chance of moving some large scale donations (think Gates) into a marginally more impactful direction.
Isn’t the point more “it’s easy to fall into the trap to believe that predicting when the sigmoid is going to bend is possible and the right heuristic is to instead extrapolate locally”?
That aside, I’d question whether applying the Lindy effect in particular to something that’s not really a life expectancy but more a growth rate is credible… or perhaps a bit circular since it “assumes away” the ceiling.
Predicting when the sigmoid bends is difficult and predicting how long until it unbends is equally difficult.
The simpler assumption is that over enough time, the S functions stack together for long enough that working backwards from exponential is a better predictor of reality.
These stacked S curves have continually been true with most technology.