Well I'm going to be different and admit that yes I do care what people think. Not people in general, but my family, my friends... people I love - I care what they think of me.
My brother, for example, is a bit of a puritan. He'd take a very dim view of SH if he knew of its existence and would lecture me mercilessly if he knew I was a member. He's a sociophobic old stick in the mud, but he's my brother and I love him, and there are things about me I'd rather he didn't know because I see no point in upsetting him and causing arguments over something that I'm never going to change his opinion about.
Yes it's my life and I'll do as I please, but there's no reason for me to go shouting it from the rooftops and antagonize people who would be uncomfortable with that aspect of my lifestyle.