First off, let me say that I am a vindictive bitch, so maybe I'll be a bit off.
I think you do. I may be a big fat hypocrite for saying so as I wear clothes my exes got me all of the time, but I feel like A) I earned it for dealing with their crap and most of their abusive behavior and B) It's completely fine if I don't tell my current partner where it's from, once you do it crosses the line into disrespect.
I feel like the way your husband told you those shorts were given to him by his ex was incredibly disrespectful. Also, there is a little something wrong with him wearing them everyday, it blurs the lines of "Oh she's in the past" because a physical reminder of her is there, every day.
This isn't JUST an argument about shorts, it's far from that. Sometimes you have to lay down the law and tell them that you really feel disrespected, I'm doing this right now with my ex because he never complements me and is constantly going on about other women flirting with him and how hot other women are. He didn't even realize it was hurting me so much. The flipside is that they will most likely think that their behavior is excusable for some lame reason or they just don't understand why it's disrespectful. You have to overlook the whole "Will he think I'm a heinous bitch if I tell him how i feel?" because girl, you gotta be in this for yourself and tell him what's what.
I hope things work out, and I hope I'm not helping to make a mountain out of a mole hill.