Spectrum analyzer can not separate Phase Noise and AM noise. the phase noise measurement option is showing the spectrum with log x-scale.
When you measure the phase noise of the free-running VCO, you need to check your instrument can track the frequency drift of carrier frequency, and also the sensitivity of high offset frequency.
In many case, the spectrum analyzer can not track the free-running VCO and shows the strange measurement result at close-in offset frequency.
If you have the E5500 phase noise measurement system, I recommend to use it. And if you plan to find new solution, I recommend the E5052B signal source analyzer.
To measure the phase noise, PLL method or Discriminator method is used. To measure the free-running VCO, you need to use the discriminator method because the PLL can not track the drifty signal source. In the E5052B, heterodyne digital discriminator is available and you will be able to measure the phase noise by just connecting the DUT. To capture 20GHz, you need to add the E5053A downconverter.
For more detail, please refer to following Agilent app notes.
“E5052B Technical Overview: See how easy it is to characterize a free running VCO”
http://cp.literature.agilent.com/litweb/pdf/5989-5040EN.pdf“Agilent E5052B Signal Source Analyzer Advanced Phase Noise and Transient Measurement Techniques”
http://cp.literature.agilent.com/litweb/pdf/5989-5131EN.pdf