How Late Is Too Late?

My husband and I have been married 13 1/2 years. We have two daughters, 10 and 12, and basically, a good marraige. But in the last few years he has decided it's okay to stay out till 4:30 and 5:30 in the morning. At first I tried to keep up with him but found I was useless the next day. Now I only hang in there on occasion, mostly choosing to end my evening somewhere between 2 and 3 am, sometimes earlier. On these nights, and some when I choose not to go at all, he often keeps going till the wee hours of the morning. My question is: am I justified to be angry with him for not coming home with me or for staying out all night when I choose not to go at all?

Important points to consider:
1. Our children are never left alone. They are always with a grandparent or at a friends house spending the night.
2. Our hometown doesn't shut down until 6am...(Casino's came to town and changed the cutoff drinking hours from 2am to 6am).
3. He doesn't gamble.
4. I totally trust him...we have both been faithful during our whole relationship...I know some of you may not believe this, but we have.
5. He doesn't do drugs. He just drinks then isn't ready to end his fun.
6. Some months he may do this a few times, while others not at all. It's random, just depending on when friends are getting together, or special occasions, and weekends only. (But more than I like)
7. He's incredibly helpful around the house and with the kids, doing dishes, cooking, cleaning, chauffering, etc.
8. This going out thing is pretty much his one main fault.

My problems with it:
1. I don't sleep well because I'm worried about him. (I have two children, I don't need to worry about a third)
2. I think because I don't like it and he knows that he is being disrepectful to me by doing it anyway.
3. It sends a bad message to people who see him out that late who know us, especially if I'm not with him.
4. Who's to say because I trust him now I will always be able to trust him. Being out in the drinking/dancing scene is an accident waiting to happen.
5. I just feel it's time for him to grow up.

Again, my question is this: Am I justified to be angry with him for not coming home with me or for staying out all night when I choose not to go at all?