as the picture shows, a photodiode receives the IR photo current, and through R1, generating a signal voltage, then, through VGA-BPF-notch-amp-comp-integrator... among them, bpf is gm-c structure with center frequency f0=20khz, Q=9, notch filter is Twin-T structure, with center frequency f0=50khz, amp is an inverting amplifier (closed loop gain=40).
now, there is a 50khz noise in photodiode, the noise amplitude can be 100mv or so, and the smallest signal amplitude can be 100uv or smaller (frequency=20khz). so, in this case, the noise frequency is very close to signal frequency, and noise amplitude is three order larger than signal amplitude. VGA has a very high gain (70dB or so), and VGA output swing is 200mv, so the VGA output is a saturated square wave (very easy to saturate), and noise and signal mixed together, signal can be drowned by 50khz noise. if VGA output is saturated, how to reject noise and 'pick up' the signal correctly?
besides, when there is no noise, and signal is small (nA order), the bpf output is basically ok, but if the signal is large (1uA or larger), the bpf output will oscillate a long time after photo signal ends, and causing wrong result. why? in simulation, i also found that, the ac and transient response is related to  my reference voltage (bpf, notch, amplifier, etc), very strange.
i have tried some methods, but have no idea yet. i need your suggestions here.
thanks.

yes, i did as what you said, but have no effect. i wonder further suggestions here. thanks.

It's probably resulting from the output resistance of the reference circuit. There is one thing though.. is the 2 pF capacitance a model of a bondpad?Then put some 100 nF on your bondpad, and it'll probably be fine.