First and foremost, DO NOT FEEL GUILTY. You are absolutely not at all wrong for feel the way you do. At 21, you are just starting out your life. PLEASE do not let yourself get guilted into staying with someone who does not want the same things you do. It's a waste of time. He will drag you down - emotionally, financially, and quite possibly, legally. Drugs are ILLEGAL. I'd hate to see you in the wrong place at the wrong time, and get caught up in something that could land you in jail, over his addiction.
I know all too well what addiction can do to a relationship (I have seen it with my parents). His addiction (whether he's willing to admit he has one or not, if he has the same routine everyday, it will soon be an addiction if it hasn't already become one), will take priority over EVERYTHING, including you, your finances and your potential future family. You deserve so much more, and you want more for yourself. Go after it, and leave him in his trailer, with his nasty roomates and his weed.
Good luck to you! *hugs*