Please look at
7 Habits of Highly Successful S-Parameters which is a good starting point.
I'm not sure what exactly your "lossy channel" is - if it's essentially a passive network (such as extracted by an electromagnetic simulator), then bbspice would be the best choice. I would start with the default parameters though rather than tightening the accuracy and trying to force fitting or anything like that. If it's not passive, then use linear (with sufficient number of points, linear would be reasonable).
You might also want to add "-cc" to the spectre command line (in ADE go to Setup->Environment and add -cc in the userCmdLineOptions field). In general you don't want this (for performance0 but clearing the cache would mean that you get any warning messages produced by the simulator when it produces the time domain model. It should tell you information about how it managed to do the fit, any problems, plus (with bbspice) the fitted s-parameters. You can plot both the original s-parameters and the fitted parameters (they can be opened as a result database in ViVA) to check if the original data is smooth, and whether the fitted parameters look significantly different.
Otherwise, if you get stuck, I'd talk to Cadence customer support at
http://support.cadence.comRegards,
Andrew.