Boyfriend controlling about friends?

Been with boyfriend 3 years. Broke up a few times for a week or so each time; nothing major. He likes to stay home and go to bed early, which is fine. But when I go out with my friends he gets mad that I stay out until 3am or so. Its not often that it happens but when it does we fight about it. I invite him to come but he says he doesn't like going out with people that drink and stay out late because they're losers. He says my friends are whores and lowlifes. He also gets mad that I don't go to church with him family. His religion is seems like a cult to me and I don't want to be involved. Is he being controlling? He says I'm crazy for not understanding his point.