That's more or less the idea behind Low-Rank Adaptation, or LoRA.

There's also Mixture of LoRA Experts, which instead of slicing up the model and routing through that, routes through different LoRAs.

But it all comes with tradeoffs, as you have to train and run the gating network doing the routing, which also comes at a cost.