Coming from a decade-long vegan: If I could, I'd line my sheets, my car, my toilet seat in fur: it's just tops when it comes to tactile sensations, but I very rarely like a fur piece visually. It's so easy to slip into gaudiness. While I wouldn't support animal-consumption, I don't think wearing fur is as near an immoral act as it's been made out to be; I think what makes people especially outraged is that
1. fur comes from "cute" animals (kindchenschema is the word for the weirdly protective feeling we have towards animals with big eyes and little heads in other species.) No one has these outraged feelings for snake skin, because snakes are not "cute" animals- same with cow leather, alligator, or shark. People relate to mammals more, making their bodies more taboo.
2. fur is the layer we actually see on animals. No one I know has ever rallied against cat gut being used in musical instruments because we don't conceptualize cat gut as an animal very well, in the same way you can't see the suede on a lamb while it's hopping around. Even meats aren't a direct link as you don't automatically associate an animal with its muscles, and their meat is often given other names depending on your language (like "pork, caviar, veal, steak, ect."). Fur is just a really easy visual translation of "that was an animal".
So if your fur is ethically sourced (i.e. second hand) and you have no problem with other animal products, I say go for it. It's a luxury for a reason, and faux doesn't begin to come close. People are irrational when their reasoning is based on feelings and not logic, and you're not responsible for their hypocritical illogical tantrum-y crap.