Even more un-fathomable, Mozilla's CTO has been writing a substack with the underlying theme of "if the AI isn't running on your hardware, you don't own it" https://newsletter.ownersnotrenters.com/

They're still 100% cognizant! And yet.

You cannot ship on device inference with a real LLM still. Average people freak the hell out if it's even 10% slower than whatever google ships. At least thats the claim for how Firefox lost all their marketshare in the first place (Which was not the case. When everyone was claiming firefox was "slow", it simply was not, as long as you used an ad blocker. Normal people like my dad didn't switch to Chrome. It was installed through a sketchy mechanism and they never noticed)

Meanwhile, if you CAN run these models locally, using your own hardware for inference is supported. It's a setting option.

Google still blasts you to install Chrome at every opportunity if you use their services, and is heavily optimized for Chrome (e.g. Docs, YouTube). That's not really Firefox's fault.

I've developed a benchmark that I think should be resistant to saturation, is easily verifiable, and anecdotally correlates with desirable behavior (ability to not get confused while generating text with state).

I think it's interesting, I think other people would find it useful, but I don't want to spend a bunch of money running it against all the frontier models.

What's the best way to reach out to labs like yours to collaborate on something like that? Are there any labs that are more open to submissions from internet randos?

SlimeBallBench? Looks cool, love the implementation! You would most likely need a LOT of environments like these if your goal was selling them to labs

SlimeBallBench was not me, but it is cool! and at least partially what inspired me to build my own.

Mine is less fun, but hopefully more useful https://erikwiffin.github.io/memory-reasoning-eval/

Selling to labs is more than I'm looking for. I'm aiming for a couple hundred dollars so I don't have to finance a Fable vs Sol run out of my own pocket. It would be cool to have my benchmark be one of the ones referenced in a model card!

Ugh, that happened to me too. But instead of researching other apps and waiting for them to decide to go premium too, I built my own workout app. Been using it for a couple months now and it's working great, but the hassle of publishing it to the app store means that it will only ever be a solution for me and I can't share it with anyone.

Have you considered making it into a PWA?

I've always thought of cloning the old Zero fasting app into one, when I have a bit more time to point my Claude at building something else.

I actually started with it as a PWA, but ended up rewriting it in react-native. I wanted a couple features (heart rate monitor, lock screen integration) that afaik don't work with PWAs.