I narrowed the problem down to my IF filter. I guess i need to improve the group delay...does anybody know how to relate group delay to ISI/BER? My guess would be to take the percentage of the power with a group delay more than 1/BR  (BR = bit rate) given the modulation scheme and consider that the SNR.

any thoughts?

cheers,
Aaron

the group delay of the filter is set by the filter response (poles and zeros) and is often plotted in filter tables (a book I find very good for this is Williams "Electronic-Filter-Design-Handbook"). More attenuation generally means more group delay variation, so classically in order of least delay variation Bessel < Butterworth < Cheby < Ellipitic.

Then a simple ac simulation can give you the group delay. Out-of-interest, why/how are you doing PSS+PXF ?
For a particular modulation you can inspect the eye diagram, and see the impact of group delay variation...

For BER you're best simulating in something like matlab (which also has a v. good filter support), with a model of your demodulator.

you can view the eye diagram in Cadence (there is a calculator function).. basically you wrap around the transient waveforms onto a time axis of 1 symbol period... From the superposition you can see the "eye" and any noise/distortion in it likely to impact your demodulation process... so you don't need a demodulator, but you will need a modulated (in your case FSK) input signal...
For example, with an ideal schematic in a transient simulation you will see closure of the eye i your eye diagarm as your IF filter bandwidth is narrowed... getting the true value with noise and alike from an eye diagram is not efficient in Cadence (transient noise etc is inefficient) and is best suited to something like matlab...