I agree with what others have posted. Vitamin D deficiency is EXTREMELY common, especially in people who do not get a lot of exposure to the sun. Vitamin D is ESSENTIAL to absorbing calcium (and phosphorous).
I live in sunny Southern California, and even I have to take prescribed Vitamin D every now and then. Vitamin D is fat soluble, which means, if taken in pill form, it must be taken with dietary fat. Non-fat milk is NOT a good source of Vitamin D unless you're having other foods with fat in them at the same time. That's one of the things that contributed to some of my periods of Vitamin D deficiency. I was only drinking non-fat milk. When I found out my vitamin D was low, I was shocked. I thought, since I drank a ton of milk and got exposed to the sun, I would be more than fine. There were other factors that contributed to my deficiency that I don't want to get in to, but the main thing I took away from that is that low-fat milk is the way to go for Vitamin D absorption.
I would, as others have recommended, get your levels checked. My prolonged Vitamin D deficiency combined with some other things have led me to develop osteoporosis. I'm 22 years old.