The settling time of the reference will be the most important. The switches rearanging the caps would be not be limiting the settling time. Design your reference buffer with a close to Q=sqrt(0.5) damping. The loop bandwidth "including" the approximation load cap define the settling time. If the supply of the buffer have some 10nH inductance to outside and some 100pF internal buffering, your first sims should include them. So he should provide good rejection.