My fiance has an ex like that. Even though it's been years, he throws a fit if I wear any sort of Vanilla scent because it reminds him of her.
I agree with the others, I honestly don't think he has feelings for her. I think all of her chastising, insults, and demands made him feel insecure and unworthy. Things that remind him of his ex may just bring bad all those bad feelings, things that he may want to keep buried.