Tired Of Being Told I'm Selfish

"ok, here goes. here is some background info
my boyfriend and i have been together for a few years, and we live together. i have a son from a previous relationship. when he brought up the subject of living together, we had variosu talks and discussions about my son and where the realtionship was going. he stated he wanted us to be a family, to play an active father role in my son's life.
i am a single mom so when he brought it up, i wanted to maek sure he understood hwat he was gettigninto and that he was sure that this is what he wanted. i haev crazy work hours so that we do not haev to find a day care that will take my son to and from school and watch him when he gets off school.
we moved into a house that just happened to be next door to a friend of his. i brought this up because i did not want our home to turn into the "homeboy" hangout. as we are trying to raise a family. i gave him the benefit of the doubt and agreed to move into this house.
during the last few months i have been getting told that he stays home with a child that isn't his and that because of that he should be able to do as he pleases.
he leaves the child in the house for a few hours while he hangs out next door.
this occurs everyday.
i am todl i am beign selfish because i haev told him that his behavior is effecting my son. my son has grown to see him as a father and now the child is asking why he loves being with his friend smore than with him.

i feel betrayed because I asked him if this was going to be an issue and he was adamant that he just wanted us to be together as a famiy ans now it has turned into my son hangigntou by himself or me and my son hanging out by ourselves whiles he is next door"