I think that this has to do with the frequency resolution of the PSD and with the fact that the orignal PSD is displayed in Volts²/Hz.
I identified this by making a two-tone simulation (using envelope following analysis) and using the PSDBB function. I was expecting two tones of 5dBm (or 0.316V²) each at the output, but I got two tones of 24.7e-6V² each instead. However, when I zoomed in near the frequency of interest, I noticed that the peak was 24.7e-6V², but the frequencies nearby had also nonzero amplitudes. I decided then to use the integ function to add up all amplitudes within a certain bandwidth. First, I divided the orignal value by (2*RL), where RL is the load resistance, to get the correct power at each frequency. Afterwards, I integrated the resulting power over the desired bandwidth. By doing that, I got the expected value of 3.1mW, which is equivalent to 5dBm in a 50ohm load.

Hence, before you apply db10 or dbm, you must be sure that your system impedance is correctly taken into account and that your PSD frequency resolution is OK. The later has to do with the envelope following analysis setup (stop time, etc).