Frankly, no. As others have said, as long as I've got my condom then I'm reasonably well protected from them (just as they're protected from me), so what does it matter what they were doing earlier?
On a different note, I'm in a swinging relationship with a married couple. We've been doing this for nearly ten years now and are very, very good friends. We progressed to unprotected sex about 5 years back after a long discussion and a joint foray to the clinic to get ourselves tested. They only swing unprotected with me and very very rarely swing with anyone else (and then only minor and protected), as do I. Thus, we trust one another....and that's the ONLY situation in which I'd go bareback outside a "normal" relationship.
