yes this seems like it can be done.... whether it can be done within the area and power budget is a different matter though.

Its usual that for a given power consumption that in the design of most circuits there is a trade off in linearity and gain, when gain is up, linearity goes down and vice versa (having said that there is usually noise included into this balancing act). This is true for mixers, and your results seems typical for a given topology at a given power consumption. If you are not making spec, then more power or a better design topology should do the trick. I don't believe your spec is fundamentally limited by process technology node.