You can find a pretty detailed explanation of the Oscillator Noise Analysis in Appendix A of the SpectreRF User Guide, see
http://sourcelink.cadence.com/docs/files/Release_Info/icoa5033/spectreRF/appA.ht... (access to SourceLink requires registration). There you can also find details about the Floquet eigenspace that is mentioned in the warning message as well as tips for troubleshooting.
Another possible method to improve the accuracy of the PSS solution might be to increase the value of the PSS option maxacfreq beyond its default value of 40*fundamental, which will force a smaller timestep. This was suggested as a solution to a somewhat similar problem, see
http://sourcelink.cadence.com/docs/db/kdb/2002/Feb/11009978.html. You might want to try if it helps in your case.