my boyfriend thinks i should limit the amount of time i visit my dad, should I?

i recently had a baby so i have been doin the stay at home thing... my boyfriend works mon-fri 8-430... and sometimes saturdays til midnight... i moved to his city so he could keep his job but i had to leave my dad who is now 3 hrs away.. im super close with him.. i try to visit him on the weekends but if i go two weekends in a row my boyfriend gets mad and says he wants me to b with him on the weekend but he is only home on sunday... its not fair bcuz his family can see us and the baby whenever because they live 5 mins away but my dad cant and i think i should b able to bring the baby over and visit my dad ono weekends... what do you think