This effect can result from a couple of reasons:

When reducing the RBW, did you reduce the span and or sweep time ? This is normally done automatically to give the signal enough time in the RBW filter to be detected properly... too little time results in a lowered signal amplitude because of inadequate detection.

another possible reason is jitter in your LO which causes a moving (albeit slightly) mixer output IF signal. using a wider RBW will capture a wider signal range (ie all your IF power), whereas narrowing the bandwidth will on register signal power in a narrower bandwidth... so you might get a wider peak at a lower amplitude (on very narrow RBW and slow sweeptime you can see if your IF is moving, if so using a wider RBW would result in a larger amplitude as you would capture all your IF power in one peak).

cheers
aw