Take a look at
http://designers-guide.org/Modeling/modeling-rf-systems.pdfThere is a section on various ways to define the gain of an up-converter. There are two basic problems with mixers, especially IQ mixers to and from baseband:
1. There are two inputs (outputs) and one output (input).
2. The baseband can be DC while the RF is AC. This makes it hard to use rms quantities.
The solution is to define gain in terms of baseband sinusoids on either side and to clearly specify whether you are defining gain in terms of total baseband power or just the power in one baseband leg.
The situation is further complicated if your input and output units differ. For example, consider the gain of a direct conversion down converter. Your input may be volts but your output could be amps. Let's assume you are defining gain from RF input to just one baseband output. If your output is differential, you still have to say whether you are talking about output differential amps or output circulating amps. The difference is a factor of two.
Also, many times the gain is normalized to the reference impedance. You must clearly specify the reference impedance. Sometimes, the reference impedances for input and output differ, especially if one side is single ended and the other is differential. Then you must really be careful about how you define the gain.
As long as you are consistent, and as long as you clearly define gain to all your team members, you can define gain however you want. In any case, I recommend using behavioral models and test benches to define your gains. That way there is no ambiguity. Another team member need only swap out the behavioral model with his/her device level model to measure gain.