take my baby with me?

me and my boyfriend fight alot and we have a newborn... i wach after her 24/7.. if it doesnt work out and i move out i will have 2 move back with my dad 3 hrs away from him... is it wrong for me to take my baby with me? am i wrong for wanting to leave and not work it out? he makes it worst because he says if i leave he will take me to court even though he has admitted im a better parent?