Agree, except the probabilities for outcomes in the structured output. I don't think you can get those for most frontier LLMs (logprobas). You can get it for open source models but not frontier LLMs.

That number is a big deal, assuming it is well calibrated. Did they talk about calibration?

I've seen them talk about it a bit on Twitter -- it seems to be fairly well-calibrated in general, but obviously you need to test it on your use case and dial it in comparison with known data for best results.

Found it interesting that a company with this sort of a valuable IP would be using internet reachable GitHub (not behind an internal network / VPN?) and a bunch of safeguards for the IP. What's to stop one disgruntled employee from leaking the entire monorepo to a competitor?