Isn't the network too shallow to be challenging to decode? One can easily produce few layer network to solve specific goal with construction techniques, so not a surprising result in my view.

What do you mean? Is there prior art? Would you call this a LORA?

My first reaction to your question (rhetorical?) was uncharitable... so, I am curious what you meant that it was "easy."

With open models delivering close to frontier level capability, I really don't see how these labs can only focus on having best model to sustain their business. Although these model's usage will grow, I think and hope that the serving will be distributed among many players.

Essentially, if SOTA models are freely available, the playing field then is on inference where people from all around the world can participate. This basically dilutes the value of model trainer like Anthropic. I can see the panic.

Don't know why are they even pursuing Gemini. Just download the Kimi, call it Kimini and serve it on your GPU. Maybe then train next architecture based on this!