Similar to me, when I have to choose randomly I look for a number anywhere around. If it's odd, one choice, even is the other.
My strategy for rock paper scissors is to always go rock, and tell everyone that's my strategy. It's more fun for me seeing how people deal with that information than the win would be. Not very random though
I don't think Zipf's law gives you this. It mostly says that common words tend to be short, but it doesn't tell you what proportion of the text will be common words. For example, in English writing two-letter words are found much more than one-letter words.
However, I think you're correct anyway. The right distribution to look at is something like the orange series in the bottom graph on [0]. Eyeballing it, it does look like 3/5/7/9/11-letter words appear more in aggregate than those of 2/4/6/8/10 (49% Vs 44.5%), and longer words are a rounding error.
I build ai systems for ops automation and don’t understand the author’s pessimism.
I agree with that Evals are a scarce commodity rn. A business needs to define what good looks like. This is a laborious, and sometimes politically controversial, process.
Given good Evals, frontier llms are a magical tool that can automate tasks and do them more accurately than human ops teams.
Exactly which llm to use depends on the mixture of speed, cost and quality of the output.
Jev makes a claim to expand some regions of the Pareto frontier. I look forward to testing if this is true.
There are many areas of work we can’t automate rn. We cannot create good Evals either because time horizons are too long, or it’s too difficult to create good Evals.
That doesn’t mean there’s anything wrong with building good ai engineering systems in areas where it works magically.
> That doesn’t mean there’s anything wrong with building good ai engineering systems in areas where it works magically.
Without the evals, how do you know it works magically? I think that’s the entire point of the authors article, people are using AI tools, and trusting the AI own evaluation of its performance, without making any attempt to validate if the AIs confidence actually correlates to anything meaningful.
From what I’ve seen, the result tends to be very exciting demos, followed by incredibly mediocre results when deployed into a real environment. The lack of critical evaluation then provides a double whammy, because it takes longer than it should to realise the system isn’t actually working correctly, and then the team lacks the skills and understanding needed to improve the system they’ve built. You just kinda have to hope that the next model release is going to magically result in better performance for your particular scenario, which won’t be able to measure due to the lack of evals.
You need a set of Evals, which catch enough of the mistakes your llm makes following your baseline prompt, so you have an equal or lower error rate than humans.
This can be determined by offline benchmarks if you build a system that takes small sequences of actions, and requires live ab test for long sequences of actions.
The more your human ops team work from documented standard operating procedure, rather than tacit knowledge, the less you need Evals except to capture edge cases
I wonder if you could pursue an antitrust cases in eu against Claude code refusing to use fable to work on some deep learning problems and downgrading to an inferior model.
Would this ever be viewed as an abuse of dominant market position, aiming to disadvantage potential entrants.
This is what I also use for rock, paper scissors (mod 3). You can also use second hand of a watch for random samples, e.g splitting 0-30 vs. 31-60.