I really feel it's a combination of the two that's very important, which is probably why it's so hard for many of us to find a decent mascara!
I've tried tons and found that if one or the other is off (great formula, sucky applicator or vice versa), it's going to suck no matter what technique I try. That's not to say technique isn't important either, it most certainly is. Slightly wiggling the want back and forth as you coat your lashes helps separate clumps and defines them better, especially if followed by a run through of an eyelash comb. Some people like to apply their mascara holding the wand vertically and using mostly the tip, which I love doing to get my baby inner corner hairs.
Overall I'd say in order of importance for making mascara work (for me anyway) is like this:
-Formula
-Wand
-Technique
You should try a few more mascaras, trust me there IS one out there that will be your HG! You gotta kiss a lotta frogs to find a prince
Personally, I looove waterproof formulas. I have thick, stick straight lashes that fall right down with ANY regular mascara, and have only had success with waterproof. I don't find them that much of a hassle because they all come off easy peasy with olive oil.
My favorite mascaras happen to be drugstore (mainly because I am poor lol) so you don't necessarily have to by an expensive one to see the differences in formula. I'm currently in love with Max Factor 2000 Calorie Waterproof and my old standby, Maybelline Full n Soft Waterproof. If you like the new bendy, no clump bristles I love Max Factor Volume Couture Waterproof.