A sense of shame is a terrible thing to have. I want you to repeat after me: "What I do on my own time has no bearing on my professional time, and if I enjoy it and it doesn't hurt people it is nobody else's business but my own. If need be, I am willing to verbally defend my right to be a person outside of my career."

Or stand up to me for why that isn't a valid opinion to hold. Either one will earn/maintain my respect.