Should I always give in to sex in my marriage even when I don't want to?

My husband punishes me whenI am not ready for sex when he is. I generally give in to keep the peace, but I hate it and I am starting to resent it. I try to talk to him aout it but that causes another fight. What should I do.