Leave or Deal With His Family?

My boyfriend and I started dating our senior year of college and it has now been 6 months since we have been official. He introduced me to his family early on, and he was ecstatic because they all really liked me. He moved into an apartment with his brother in the city, because he was starting a full-time job. At this point, my boyfriend and I had been inseparable. One night, we had a huge fight (which got a little insane) and he told me that he wanted to spend less time together. He told me other things that he had problems with when it came to our relationship. It came as a shock, since he always said he loved just being with me, but we ultimately decided to take a step back and spend less time together. The biggest issue is that he told his entire family his issues with me, but he never told me until our big fight. How was I supposed to fix things when I had no idea he had problems? Ever since they found out about the fight, they have absolutely hated me. His brother has even taken away my boyfriend's phone when he found out we were talking.

They keep telling him they need to talk to him, and each time, they tell him to break up with me and say bad things about me. A couple times, he has even broken up with me saying he loves me but that it is becoming too much. We have always gotten back together, but I don't know how much more I can handle this. They said they used to like me but hated how much time we spent together, so we fixed it and hung out with our friends more. Now, it's because they think I'm a terrible person and they will not leave him alone. One thing I have never wavered on is wanting to be with him, but is it worth dealing with his family when they keep attacking us based on only negative things they heard? If it is, what should I do?