I really enjoyed reading this, got invested in learning about this idea, and then realized they they never actually show any evidence that this works? Like the text-based model they are experimenting with did a kinda counterintuitive thing that they thought was worth a section, and then they don't say whether or not it was right? The idea of aggregating chatbot context into a probabilities is super interesting and I really want to know if it's bullshit or not
I think that was the entire premise of the CIA analyst framing - you don't know which one of several outcomes is going to prevail but you have a model which constantly updates its priors to increase the probability of one outcome over others.
In this case, you still don't know whether some guy's passing percentage stabilizing over a baseline will help you win the big prize or not but it does tell you that he may win some sort of recognition for being a standout player or that the guy who scouted him deserves credit.
