You're going to have to revisit and re-specify each block.  

First, figure out what is limiting your speed to 50 MS/s?  Is it the clock generator, settling in the MDAC?  The speed of your comparator?  Until you can answer that question it is hard to give suggestions!

Also, depending on your process you have to ask yourself if 100 MS/s is do-able.

130nm is fine for 100 MS/s.  So what to change to speed it up really depends on your specs.  If you can deal with more DNL and INL, sure you can reduce your cap sizes.  You can also increase your OTA current if you need more speed.  Watch out for reduced open-loop gain in that case.  It's possible you'll need to redesign your OTA but you wont know until you really analyze your specs.  If your gain and cap sizes are already set to get the accuracy you need, you'll have no choice but to re-design your OTA for more GBW.  That's going to cost you power, but there really isn't a way around it.  It *may* be more efficient to put in a digital calibration system to relax the interstage gain requirements, but that may be more trouble than it's worth.

As for the references, there are two schools of thought there.  One is to design a reference buffer so fast that it settles in half a clock cycle.  Another is to put a big enough off-chip cap on the reference that it doesn't move more than a fraction of an LSB when you are operating.

Personally, I've had better luck using an off-chip cap.  Just be aware that now you have a resonant circuit (including bondwire or bump inductance) in your reference path.  You must simulate it thoroughly (post-layout) to make sure it doesn't peak at a troublesome point or even oscillate.

I don't recommend using a completely off-chip reference... it makes the board design harder because any interference on your reference goes right to the output.  I would recommend putting a bandgap on the chip but be sure it can be bypassed if it fails.