Yutao,
The experiment you tried is interesting, but still leads to a few possibilities causing the problem. Does a simulation yield the similar results?
When you injected this signal into the rx port, was it still connected to the TX port through the switch?
If yes it is so then it might be the mechanism of coupling you state or at least a similar mechanism. but since the vco is running at 2x the LO then its probably getting on the divider possibly through a common supply. This mechanism once found should be an easy fix.
if no, Does the experiment you tried yield similar results if you remove the rx path matching circuit completely. If so then I would be more inclined to believe its the transmitters reflected signal causing the imbalance, hopefully this is not the case!
For the yes case
Using a high impedance probe (or a simple stripped coax with a DC block to act as an antenna) and a spectrum analyzer probe the supply for the VCO by injecting the a tone into the disabled rx path. Also look to see if you get strong relative sidebands around the VCO and 2xVCO this would indicate that the its on the VCO signal. Record
relative levels of all spurs, and try to reproduce them in the simulation by introducing them one at a time until you see in simulation the same as in measured.
Good luck
http://rfcooltools.com