Why does he still insist that i am being too unfair?

My son is 7 1/2 years old. His father left me when i was 3 months pregnant giving me the ultimatum- it's me or the kid.... i chose to have the baby and told him where to go, we were together for 3 years when this happened. As soon i had my son, i can not fault him, he is the best dad that any mother could ask a child to have. The only problem is this, I have done all that I can to keep my son's relationship with his father consistant and fair..... maybe i have been too fair??? He only pays me $50 child support a week, (He is supposed to pay $120)- my suggestion as he spends 2 days a week there... fair???, If it is a public holiday and his father rings up and wants to see him and i havent already made plans with my son.... he goes..... fair???, i drop my son off to his place and also pick him up.... fair??? Last weekend was the fathers weekend, but asked me to keep my son home as he was going out nightclubbing, that is fine, but this weekend was suposed to be my weekend and i had plans to do things with my son, but because his father chose to go out on his weekend with his son, now feels like he has missed a weekend and wants to swap, which would have been fine, but we already had plans. Now that things haven't gone his way he is threatening me with all sorts of things like going for custorty and not paying child support, etc. I can't understand why he is being like this and am sick of the constant battle, it's not like im one of those spiteful mothers that use the emotional blackmail of kids to get to the fathers emotions and make them angry!! Have i been too fair and now he just takes it for granted??? Im sick of being so nice to make peace and just been treated like crap... what should i do???