These are all surely FAQ type tips for PSK and other sound card based digital mode ops on the ham bands, but there is so much mis-information out there and confusion about the details that I figured I'd give it a crack and word it myself. The hope is to aid in clean signals and smooth ops out there.

Controlling noise on the sound card interface

Proper sound card interfaces should have RF isolation in them, this just means that you take the audio lines and run them through 1:1 isolation transformers. This ensures you don't have ground loop noise or RF on the audio lines. Most commercial interfaces have this and with the least expensive options like the Donner, this is basically what you are paying for.

Rig output power settings

Any HF rig should be run at full power out. Set the SSB power out for the rig to 100W or full power for your particular rig. The rig itself should be run full open at 100W to maintain the best linearity of your signal.

Rig audio considerations

Ensure that there is no speech processing, RX EQ or TX EQ enabled at all while in digi mode. You do not want any alteration of the tones being sent on TX or RX whatsoever. Even small amounts of alteration will affect the ability to decode what you send and also what you receive. This also goes for noise blankers and noise reduction filters. Some modes are tolerant of a bit of either being applied but pay careful attention to learn which ones tolerate it and what the effect is on your receive.

Rig AGC suggestions

If you have the ability to do so, adjust your rig for a fast AGC, though this will vary in implementation, do experiment with the results you get with fast AGC set. On older rigs with lower option levels, you may have better results with the AGC off entirely and riding the RF gain to get the cleanest appearing signal on the waterfall. Getting this right can make a huge difference particularly with marginal band conditions.

Transmit Power configuration.

The way you dial in your desired TX power level is as follows. Start with your rig at full power, but your TX audio level on the computer set to zero whether it be on the front face of a Signalink or in the software mixer of your operating system. For people using their mic jack for input, set your mic gain ahead of this step so that you have actual 'proper' ALC about 1/3 to 1/2 of the proper ALC range when speaking in SSB mode. Do not set it with the mic gain mashed to the right or you will have trouble. Take your digi mode software and start it transmitting in 'tune' mode. In fldigi that is the top right-most option button. While watching your power output meter, on the rig or on your tuner, whatever you have, very gradually bring the level of your TX audio up as you watch your meter. This is how you control your TX power and will get the most linear and clean signal this way. You want to see NO ALC indication on your rig with the meter in ALC mode. ALC indicates compression, which distorts your signal. If you do this properly, even the oldest and cheapest rigs should deliver right up to 100W without any ALC deflection indicated at all. If you are seeing ALC at low power levels such as 25-50W which is much more advisable with digi modes, you have one of the steps above out of whack or some other technical issue.

Receive audio configuration

This varies from band to band and the conditions, but it is important to note the receive level audio indicator on your software so you are getting enough signal, but not overdriving. In fldigi this is the small diamond just to the left of the AFC button in the lower right corner. Adjust your RX audio coming from the rig so that you see this diamond blinking yellow to red...then back it off slowly until it no longer blinks yellow or does so very infrequently. If you overdrive the incoming audio, it will clip and cause errors decoding signals.

Comment on power levels in general

With PSK in the standard areas of the bands, you will be along side of, at times, nearly 100 ongoing QSO's, shoulder to shoulder. As a courtesy, most ops stick to 25W or less to avoid causing AGC pumping interference to the other operators. Please keep this in mind. Too much power among many weaker signals is what causes the waterfall to 'pump' and you to lose decode on signals. Outside the normal ranges say +/- 1kHz of any group of PSK signals, running whatever power you need to make a contact can be reasonable. MFSK modes and MT-63 benefit from this more than PSK does. With the high duty cycle of MFSK/Olivia/Domino/Thor/Contestia with most 100W rigs it would be advisable to run no more than 35W for extended transmissions. I personally only run 25W from the TS-2000 as it gets quite warm at that level. If your rig has an efficient fan or is rated for 100% duty cycle like some Icom rigs or older big contest rigs are, you can probably work at 50W if needed. If you add a fan on the heat sink, you can work higher powers a bit more comfortably. I still advise to take it easy, a good number of people have cooked their rigs running 100W on digi modes and have cooked transistorized amps running them at more than about 1/4 of their ratings without a fan. With MFSK modes it is a fairly common courtesy to have 500 kHz spacing between any other ongoing QSOs to give some breathing room, particularly with using higher power levels. This is also a nice amount of headroom to allow for any ongoing CW QSOs as they share common space on many bands. Keep in mind however that in general, if 50W isn't really doing the trick, the band is in such bad shape that 100W won't help you. As you try to mash down on the power, you are far more likely to put out a nasty signal as well as cooking your rig finals.

Comment on duty cycle

Many of the digi modes, PSK type, MFSK types, are quite high duty cycle, like the RTTY ratings that manufacturers used to issue. Beware of hammering your rig for multi-minute long transmissions at high power in digi modes without some needed cooling off time or an adequate fan on your heat sink. It isn't that hard to ruin a rig or amp with overzealous digi ops.

Comment on using a linear amp for digi modes

Normally high power QRO is not needed at all with digi modes. Even with an AL-80B here in the shack, I cap my digi mode ops at 100W and only for shorter MFSK mode QSOs when I really need to get the copy through or when operating Feld Hell (very low duty cycle, less than SSB). However, you can use a linear to ensure cool operating during long winded MFSK based QSOs. It can be hard on your rig to chat for two hours using Olivia but I do it frequently. Here is how I tune and apply using my linear for digi mode ops. Select your dummy load.... Set your TX audio level to zero, rig should be at 100W SSB. Very slooowly bring up the TX audio level until you see a few watts and enough indication on the amp meters to begin setting the plate and load. Continue gradually bringing up the TX audio level as you tweak the plate and load, finally coming to proper max power out with a properly loaded amp. Basically as you would with a CW tone during normal tuning but using the TX 'tune' sequence in your digi software. Now simply lower your sound card TX audio level mixer until you see the desired output from your amp. Do NOT re-tune the amp for the lower drive level. I very typically run just a few watts from the rig into the amp to get 50W out for evening, low-band, rag chewing with Olivia. I can easily run less if the band conditions are solid and quiet. This gives me the maximum linearity from the rig and the amplifier and I've never had anything but pristine signal reports following this approach. I can chat at very slow Olivia 8/500 or 16/500 rates for hours with transmissions that take several minutes to complete with both the rig and the amp running optimally. Be very careful to follow this setup, if you load your amp up at low power you will likely have a horrible signal result. This is actually exactly how you should run your amp for lower drive output SSB use too. Many hams tune up at low levels and wind up overshooting the loading causing a really dirty signal. If your amp has an ALC control like the AL-80B, just as with the rig ALC, you should not have any engagement of ALC whatsoever for digi mode ops.
I hope this has been helpful and I encourage pointing people to this link for help when needed. 73 de Chris KC2RGW