I tried to increase the current of LC QVCO tank to increase the output amplitude that drives buffers.
I found  the tuning range looks odd. Near to the output common-mode (around 1.3-1.4V) it starts to have a dip & hump.  The lower range is saturated.  If I reduce back to the current, it looks ok and have wider tuning range.

looks like you are not sweeping the control voltage in fine enough increments.  That's why it appears non-monotonic.  Add more points and it will smooth out --- and then probably tell you it's still non-monotonic due to a model or change in region of operation.

hmm.....interesting.

Does the Q of the varactor show a Dip at CM voltage ???
If possible, pls. post the graph of Qvaractor vs. Input Current.

if it were the case, then i doubt it would show the effect you originally quoted. Quote:
It should have shown same behaviour approaching from both sides.

Have you checked Raja's point whether simulator converged properly at those spiky points ?

How does the voltage amplitude vary with frequency (does it for instance peak around where you see the kink in the resonant frequency curve) - can you post a picture of that?

I also seem to recall that with cadence, using the low accuracy presets does funny stuff like this - try running a simulation with the conservative setting.

it was prelayout sim

it is not due to accuracy setting.

Highly suspect on the Q of the nmosvar.
However I could not simulate Q nmosvar because it gave me bogus results!

see thread
http://www.designers-guide.org/Forum/YaBB.pl?num=1274702712/4#4

The Q normally is the lowest at common-mode and the weird behavior always happens near to the common-mode.

I have reduced the length of nmosvar, everything looks fine.
In the previous setting, the length is about 3x of min to increase tuning range.

found that the Q of the cap array is pretty small after knowing how to extract Q using ac method rather than sp analysis.

improve the Q of the mim cap array . Just wonder why the Q is pretty small in 5GHz. How could it work in >10GHz application?

I don't think Q can cause f-v curve non-monotonicity. Q can cause amplitude change whcih could cause a change in f, but thats usually not a very big effect.