am i wrong to not let my baby's father see his daughter?

see my daughter was in the hospital for 5 weeks because she was 5 weeks premature and he only came to see her once for five mintues. since she's been out he has seen her 3 times and she's four months. he wasn't there to sign the birth certificate and he doesn't pay child support. he still uses drugs and i don't want my daughter around that. he keeps calling me saying my daughter will hate me because i want to keep her from him.