Eugene

Finding a detailed example of how to specify an RF front end will be hard to find since details are usually proprietary. However, there are some high level papers and some general text books available. I list those at the end of this message.
The main problem is in transforming DSP metrics such as EVM, PER, BER into RF metrics such as IIP3, noise figure(NF), gain, SNR, and bandwidth. Typically the systems guys perform Monte Carlo simulations to generate BER(SNR) curves. From the curves and the target BER, they determine the minimum SNR required. From there, the RF engineer starts selecting the number of ADC bits and noise figure. Adjacent and alternate adjacent channel interference will determine the filtering requirements and IIP3. The RF engineers usually work out the link budget. The link budget distributes the IIP3, NF, gain, and filtering along the receiver chain. You can find short courses on how to perform link budgets. Link budgets are usually performed on spreadsheets. Again, the trick is to relate endtoend RF metrics to DSP metrics. One popular solution to checking BER is to build up a model library of RF blocks in the DSP design environment. With a candidate RF architecture, the systems team can simulate BER. To simulate BER, you must account for RF impairments as well as the DSP remedies (equalizers, encoders, acquisition algorithms etc.) The RF models should also account for frequency offsets, DC offsets, and phase noise. At this level the models do not have to be too complicated. The models should suppress the carrier lest the simulations run on geological time scales.
Here are some references that may help:
"A 5GHz CMOS Transceiver for IEEE 802.11a Wirelss LAN". David Su, Masoud Zargari, ..... All from Atheros. 2002 ISSCC
"An Integrated 802.11a Baseband and MAC Processor". John Thompson, Bevan Baas, .... All from Atheros. 2002 ISSCC
The Atheros papers are high level but they describe a superhet 802.11a design.
"Behavioral Modeling of Nonlinear RF and Microwave Devices". Turlington. ISBN 1580530141
Turlington's book describes spreadsheet methods.
"CMOS Wirelss Transceiver Design". Crols and Stayaert. ISBN 0792399609
This book describes a GSM CMOS example in detail.
