Period jitter should be reduced as the phase noise is reduced in the PLL design. However, what I got in the simulatin does not prove this.

I rebuilt the PLL behavioral model according to the Ken Kundert's paper "Modeling jitter in PLL-based freqeuncy synthesizers".

With only noise in VCO, for the VCO open loop and close loop, I got exact the phase noise as shown in Figrue 9 in the paper. However, I calculated the period jitter of the VCO output, which is the standard deviation of the VCO output periods recorded in the periods.m. The period jitter is a little bigger in closed loop than the open loop. In closed loop it is 5ps while in open loop it is 4.5 ps, same s the injected jitter in the VCO model. The jitters here are JFD. They can be converted to Jvco by divided sqrt(N) as described in table 3 of the paper.

What I got is period jitter becomes worse while phase jitter is better. I am sure the model and simulation are correct since the phase noise matches the result in the paper.

Here are my guess:
The noise model is not good for time domain since the noise added in VCO has unlimited (or too big) bandwidth. In real world, this is not ture.
What do you think about it?

And what is the right way to model jitter in VCO to simulate in time domain, so the period jitter of PLL can be predicated?

When measuring period jitter you are only looking at the signal for the duration of one cycle, so you are only really seeing high frequency noise. The loop has finite bandwidth, so you should not expect it to reduce noise at high frequencies. Your results are consistent with what I would expect with period jitter.

-Ken