I tend to just avoid a group if I find they are unpleasant without letting that taint my view of the media that they are a "fandom" of. I'm sure that for the vast majority of popular media there is a much larger group of people who enjoy it on their own without participating in these fandoms anyway.