Should I give up on my relationship and move on or stick it out?

This is my ex-husband and we have been trying to rekindle our lives together. We have a 16 yr old girl together. I also have 2 young great nephews whom I am in the process of adopting. Everything goes well as long as I let him do whatever he wants. Generally this is not a problem. Unfortunately we do not live together and we have been battling this. The problem is he has his friends come over and spend the night whenever they need and he goes over there houses and spends the night. There is always drinking involved. Which renders them unable to drive. I do no promote drinking and drinving. But I do not feel it is necessary to have this around my children. Please give me some advice. It is split down the middle on what to do. I love him dearly but don't want that kind of lifestyle. As long as I give in to him he is fine. But if I tell him how I feel it turns into a huge major fight and he doesn't give a care.