After looking at freely available SVGs of pelicans and bicycles, I have a hard time imagining what they could be using to game this.

Then maybe what you want is email?

Sprinkled with a little bit of https://delta.chat ?

And potentially a dash of iroh https://news.ycombinator.com/item?id=48542480 https://www.iroh.computer/solutions/delta-chat ?

Delta.chat no longer uses email though, right?

delta.chat is email

https://delta.chat/en/help#can-i-use-a-classic-email-address...

Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...

I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?

Nothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks.

https://machinelearning.apple.com/research/introducing-third...

Thanks for responding.

Could it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?

I would wager not a lot. There are some real hard constraints in space, from power consumption, to weight to thermal output (lack of convection is a real PITA for thermal shedding), and the list goes on.