There are market verticals where this makes a lot of sense. Embedded systems and IoT devices comes to mind.
Even in data center space, I believe several layers can use fixed weights and remaining layers will compensate for the variations. Power savings will be huge so I think there is an incentive to do more of these.
Even for consumers.. Every enthusiast now wants those specced out rigs to play with LLMs. Make a nice chip for that, and it will reduce some pressure on consumer RAM demand
Even in data center space, I believe several layers can use fixed weights and remaining layers will compensate for the variations. Power savings will be huge so I think there is an incentive to do more of these.