you obviously need to do some work on scaling the integrator states and get the quantizer input to come down so as not to overload your 8-level quantizer. if the quantizer input is too large (assuming a fixed number of levels), and the quantizer output is hard saturated at the limit, then your quantizer gain drops, pulling the loop into instability.

whatever reference you are using does not mention the dynamic range scaling and integrator state limiting that was performed on the modulator.

regards,
Vivek

Well, you don't really have any other options, do you? Scaling integrator levels is a necessary step in the design of any filter, and you want to do this to
(a) limit the swings of the various integrators to a reasonable level
(b) make the swings somewhat similar, e.g. not that Int1 swings from -1 to +1 and Int3 from -1000 to +1000, which might easily happen without
dynamic range scaling.

in addition, you want to add some saturation in there. the common practice is to set saturation levels so that the integrators later in the chain saturate slightly earlier when using a CIFF topology. Once this is done, the quantizer input swing is automatically defined, and you can decide what number of quantizer levels you need.

The common practice though is to first decide on the number of quantizer levels you can afford to have, and then perform the scaling so that you do not overload your quantizer.

If you want to know more, you will need to read up some books, e.g. "Understanding Delta-Sigma Converters" by Schreier & Temes, or the yellow book by Norsworthy, Schreier and Temes, both free for download at IEEE Xplore under "Books" if you are an IEEE member.

Vivek