...depends on the bandwidth of the system you are looking at. Generally one is interested in the in-band effects of two tones, so you should ensure they are close enough together to enusre inband response.. but usually far enough away to improve simulation time. e.g a if the bandwidth is 10MHz, spacing the tones at 1MHz or 5MHz should give the same result (any difference may be due to small effects like passband reponse ripple or LO phase noise etc.) but 5MHz would be qicker to simulate!

The bottleneck could be either the input or the output. Depending on your system lineup will depend on whether you care about output compression/clipping. You may find the following stage from the mixer may have an input compression far lower (6dB or more) than your mixer's output compression so you shouldn't care about output clipping in that case.

Anyway to see whether output clipping is the problem, often I artificaly increase the power supply to allow the output enough headroom to swing freely, and see whether the compression eases (of course in conjunction with hand calculations as to what the linearity should be based upon device overdrive etc.). This simulation approach will be especially effective given you are resistively loaded (up to vdd i guess). If using gilbert style mixer, one should take care that DC bias of the commutating state is optimised.

Ways to increase input compression without decreasing gain (or increasing noise) is problematic unless you have more current. Typical methods as well as increaing over drive are degeneration (either resistive or inductive) or doublets/triplets (offset pairs the same as multi-tanh in BJT technology). I have used both these methods and found both have advantages and disadvantages, but both work well. There are other ways to do this, most RF books (like Tom Lee's "The Design ofCMOS Radio Frequency Integrated Circuits" deisgn book).

Anyway hope this helps.

The optimum DC operating point for the LO quad gates will depend on the drain (output volts drop across the load) and the operating point of the gm transistors (btm tier). For any given situation I usually sweep the DC operating point to find the optimum situation.... so i don't mean that the gate bias should be below the output bias point, as for example this can cause the input gm transistors to come out of saturation.
Anyway when you say you decreased the current in the PMOS bleed to improve the P1dB, the gain decreased right ?
cheers
aw