I cant really advise you much, i just wanted to tell that i know that damn feeling when something hurts in your chest so bad that you think its the end. I would say yes, do leave him, cause thats what men say - i messed up, im a human being, i make mistakes etc. But well, like girls before said - if it happends again it gonna hurt 100 times more. You're his partner, his girl, not a nanny to babySHIT him and keep him away from hooking with girls. Plus, you cant be around 24/7, what, you gotta worry every time he goes away? I think its not a man you can spend your life with... How long have you been together? Had he ever acted shady before?
You should worry about yourself more. This things happening will slowly drive you insane every time he goes away. You gotta at least be able to trust you man. Im sorry you're going through this. Lots of hugs & positive vibes.