Even if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who play against other humans.
I wonder why they didn’t throw a real chess engine in there for a baseline. There are engines where you can set the elo in the settings, so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other.
This is true, but I'm not sure it matters? I was poking around at the lichess database recently and those elo calibrated bots are
remarkably well calibrated, their rating variance sticks out like a sore thumb compared to human players even at similar game volumes. So it should still be a decent predictor of how good a human at that level is, even if the playstyle seems alien.
This is simply blatant misinformation. If you play a game online on lichess and go to the analysis board you can find when your game becomes novel. It will be within 20 turns unless you are intentionally following a known opening. In fact it will likely become unique within 10-15 turns.
It's not my experience at all. If you find yourself in a novel position within 10-15 moves it's likely a resignable one.
Edit: maybe you don't understand what i mean by novel position. I mean any position that has never been reached in the billions of lichess games, including bullet games among beginners.
Also yes I will concede that it's possible to make "quiet" moves. pawn nudges that barely affect anything. If you're doing those you're not 1500 ELO. You're intentionally trying to throw wrenches and I just don't see why an ELO bot even needs to bother with nonsense like that. This is supposed to be for fun / training!
