Your intuition about the maximum frequency can be misleading because in a circuit simulator, the models are split into resisitive and capacitive pieces. The resistive pieces have infinite bandwidth and can occasionally generate high frequency signals that the simulator must properly handle but that you never see externally because it gets absorbed by the capacitive pieces. Whether this is important I don't know. The easiest way to find out is to simply keep increasing
maxsidebands. However, realize that there is a point of diminishing returns: as you increase the number of sidebands the noise will keep increasing, though slowly, while the simulation time increases more rapidly. See Tables 1 & 2 in
http://www.designers-guide.com/Analysis/sc-filters.pdf to get a feel for the tradeoff.
Also, realize that simulating the stages separately will never give the accuracy of simulating them together. Off hand, I can think of two reasons for the difference. First, the driving and loading environment of the gate is somewhat different when simulated alone. Second, this particular method of measuring jitter looks a the noise at one point on transitions, but is that the best point? Also, is one point enough? Clearly, the sensitivity of the noise from a previous stage decreases as the input voltage moves away from the point of maximum gain, but how sharp is the decrease? Presumably noise from around that point also affects the output. The effect of this is captured naturally by the simulator for all but the last stage in the simulation. Thus, when simulating 5 stages together you get 1/5
th the error that you get when you simulate the stages individually (though there is a simulation procedure you can use to substantially reduce this error).
-Ken