The best way to measure image rejection is with a PXF analysis.
FFT routines are generic, they all provide the same level of accuracy. If someone is having trouble with the accuracy of the FFT results, the problem is not with the FFT, but rather the underlying transient analysis. One needs to follow the guidelines given in my book,
www.designers-guide.org/Books/dg-spice/.
Using PXF avoids the FFT altogether.
-Ken