Did you provide a small-signal stimulus?

-Ken

You didn't answer Ken's question.  Having an ac analysis doesn't mean that there's actually an ac source (small-signal stimulus) in the circuit.  If you don't have a stimulus, then you'll always get zero output.

The following is a 5V dc source *with* a magnitude=1 small-signal stimulus.
V1 1 0 5 ac 1

The simulator takes the (symbolic) derivatives of all the expressions you used in your module and uses them to determine the ac response.  It needs the derivatives anyway for using Newton's method to solve the dc operating point.

I believe the simulator is always able to compute the symbolic derivative.  Even if you have an iterative solution, it's all composed of assignments:

//dx_dv = dx0_dv;
x = x0;
for (...) begin
// dx_dv = dx_dv + dupdate_dv;
x = x + update;
end

where the //commented lines are added by the compiler generating the C code from the Verilog-A.

I don't think there are any books (nor even papers) on how Verilog-A is converted to C.  There might be something about how Verilog-A equations are translated to equations and unknowns in the simulator in "The Designer's Guide to Verilog-AMS."  (Chapter 3, Analog Modeling, Section 2.1, Conservative Systems)

As for how the simulator runs analyses, "The Designer's Guide to Spice & Spectre" is probably going to cover it for you.

If you click "Books" above, you can get to some links that will chip a few cents (maybe a buck or two?) back to help support this site.

For your actual problem: can you try writing an explicit function for the drain current (maybe the "initial guess" for the iteration)?

Also, check your if() statements carefully.  Sometimes people things like:
if (x < -100)
 Iout = 0;
else
 Iout = f(x);

forgetting that Iout = 0 also means dIout/dx = 0, and this would give you a zero output in ac.