When is it time to call it quits?

I have been dating, and now liviing with, a man 21 years younger than I am. We have been together for 2 years and are both professionals. However, we have hit a plateau with what we have in common. I prefer more relaxed activities and he loves to go out with his friends and drink till he can barely drive. Because of this, we rarely do things together. We dine out together, we take our dogs to the dog park and swimming in the lake. We go out to eat with my friends because he gets along with them. However, I do not like his friends that "party" all of the time. Alcohol and other things. He doesn't seem to be interested in tennis, golf or anything "normal" that professional men do together. I am also afraid that his partying with his immature and lack of responsiblity friends will one day effect or even end his career. I do not enable, but I also don't "have to" deal with it and live with this behavior. I thought it was our age difference, but it was pointed out to me that this behavior would be intolerable at any age. When he is not with his friends he is a good guy, very lazy and a spoiled momma's boy, but he tries. However, these friends text him all day and night. Ok, I would love to know your thoughts.