Issues with sex and smoking pot!

I have been married for 15 years and have two wonderful children. My marriage has had many bumps in the road along the way. My husband and I have an issue regarding smoking pot. My side..I hate it. His side..He likes it and uses it and wants to continue to do so. I am very much against it. I discovered that he had been using it at out family home behind my back. This has created issues with trust once again in our relationship. He cheated on me a few years back and I have never gotten past it. The woman ended up pregnant by my husband and got an abortion. I probably should have moved on at that time, but was so blind sided and ended up in a deep, numbing depression over it. My husband wants me to agree to a minimum of 2 days per week for sex, which I am trying to uphold, although having difficulty in doing so. I have alot of fear about going out on my own away from him. I have unfortunately placed all of my eggs in one basket, and he is the primary bread winner and I work part time and am the primary parent of the children. I need some opinions of outsiders. Please be sincere and kind with your response. Thanks.