Not necessarily. If your weights have to go across 2 banks, you just have to split and transfer the input and output vectors, which are much smaller.