Should I stop hanging out with my married co-worker?

Everyone thinks that I should stop hanging out with him because he is married. However, we have tooooons of fun together. I introduced him to another friend (female) of mine and she loves to hang out with him too. He does talk alot about his marital problems and once he asked to date me, he would leave his wife and kids. I squashed his advances and it has never come up again. We still have tons of fun. He wants to go to Las Vegas. Should I go? Should I stop hanging with him?