Is it wrong for me to not want my boyfriend & his ex or her family to have ANY communication??

When my boyfriend & I got together, I made it clear that I didn't approve of him & his ex still being friends. They don't have children together, so I really don't see the point. We've been together going on 3 years now & she still manages to creep back into his life every once in a while, whether it be the holidays, a funeral, or just for the hell of it. She's been told by several people (mutual friends, my boyfriend & even ME) to just stay in the past & respect that I don't approve of their friendship, which she agreed to. She's married to the guy she left my boyfriend for & just recently had a baby, but lately her older son is hanging out with my boyfriend's son at least once a week whether it's in the neighborhood or my boyfriend's mother's house. It makes me feel uncomfortable because I feel like he's still not over her & vice versa. I let him know about my discomfort & he blew up me to the point of telling me if I don't like it, I can leave!!! It kills me that he's will to battle me like a stranger in the street when it comes to her/her family. I know I should just leave but I love him dearly & can't bring myself to do so.......He says I'm being petty & there's nothing wrong w/them being friends.....