my husband wants to see his daughter- do I let him go spend the weekend with his ex and daughter?

He has not seen his daughter for three yrs. His ex say's he can meet her and the daughter for the weekend, but without me, she does not want to meet me. I do not like the Idea of me getting pushed aside, however I do want him to see his daughter.