Trust your intuition-- maybe he didn't technically cheat, but there's something "off" here. Though really, it might just be your unresolved feelings from before.
You might want to move on to another relationship that doesn't have such a rocky foundation... because if you try to make this work, both of you will be raking yourselves over the coals for a whiiiile.