Having managed to find something to open your netlist (which is in a proprietary archive format; better to use tar), the problem is that you are simulating a 2 tone circuit (13GHz LO with 12GHz RF input) in PSS using harmonic balance, with a PSS fundamental of 100MHz, with 300 harmonics...
Code:pss pss flexbalance=yes oversamplefactor=8 fund=100M harms=300
+ errpreset=moderate tstab=800n annotate=status
Not really a good choice for setup. In general I'd never use PSS in harmonic balance mode with multiple tones (OK for shooting). For a start, the simulation is very slow - and even in this case you're including only 2 harmonics of the LO and RF signals - which isn't very much!
If you simulate with the hb analysis:
Code:hb hb fundfreqs=[13G 12G] maxharms=[5 5] autotstab=yes autoharms=yes
(note that the autotstab came up with a tstab of 3.846ns but actually stopped the tstab after 2.385ns (roughly) as steady-state had been reached. The autoharms came up with 5 harmonics for the first tone. This ran in 1.1s - so was very fast. The results look good to me.
If you're using an older version, you could use qpss (in harmonic balance mode), but if you do, please set the fundname on the two LO signals to be the same - currently you have:
PORT2 (net58 net57) port r=50 type=sine freq=floQ dbm=ploQ sinephase=90 \
fundname="
FLOQ"
PORT1 (net30 net29) port r=311 type=sine freq=frf dbm=prf fundname="FRF"
PORT0 (net28 net27) port r=50 type=sine freq=floI dbm=ploI fundname="
FLOI"
Since QPSS works on frequency names, it will see these two as separate. You don't want them to be simulated independently, so give them the same fundname (frequency name) on the schematic - which will mean that QPSS sees two tones. Then set 5 harmonics of each.
The "noise" you are seeing is numerical noise. Because the solvers have to solve equations to within a tolerance, they are not exact - and the tolerance levels will result in a certain amount of numerical noise. Simulating unnecessary harmonics is going to make matters worse.
BTW, I also did try running with shooting PSS - (same unnecessarily low fundamental, and same tstab), and you will also see numerical noise, but it runs somewhat quicker than HB with 300 harmonics (and you're unlikely to be missing anything since the number of harmonics is primarily an output parameter with shooting). Tightening errpreset to conservative does lower the numerical noise floor - but you would have been far better off just having a fundamental of 1GHz...
Regards,
Andrew.