There might be serveral reasons:
1) Spectre needs to re-compile the VerilogA every time, you may follow Geoffrey_Coram's suggestion to check.
2) Some models need much calculation time. For example, if your Verilog-A model are basing on bsimcmg, then it should be slower than bsim4.
3) In most situation, the build-in models (bsim4, psp) of Spectre are optimized in internal nodes' connection. For example, if rs and rd are tiny, the intenral d/s node are shorted with external d/s node, but for Verilog-A model, most likely that a 0V vsource (V(d, di) <+ 0) are used, or a small resistor (1e-3) are added between internal and external nodes. So the build-in models need less iteration to get convergency (in fact, it needs no iteration when no internal nodes), while Verilog-A models needs more iteration.
4) The Verilog-A compiler in Spectre may be not so good. For example, a VerilogA bsim4 might be 3X slower than a build-in bsim4 in tran analysis. And if the compiler can't distinguish charge calculation from current calculation, the performance is more poor for DC, which may up to 6X slower.