is it ok 2 b upset when my fiance allows his son & his ex 2 run our lives & him say hurtful things when angry?

OK, I'm engaged to a great guy who has a child with another woman (never married, she just got pregnant on purpose after 6 months of dating) he stayed with her until their son turned 4 when she finally broke it off, it was very mutual but they still can't get along. I knew about his son and ex when we got together, but i should've stayed out of their situation. i was a child of a broken family and i didn't want his son to suffer. it's been a year and a half and we're now engaged. Being I was THAT child a long time ago, i convinced him that he needed to relax and not stress out about her witchyness so much because his responses would only hurt him and his son. I guess he relaxed TOO much. Since then, she has thrown away my Parent Football Shirt for their 5 year olds flag football team because the back of mine read "'s Lil Momma" and his EX threw my shirt in the trash. What she doesnt know is that I've been called Lil Momma since MY DAUGHTER was born!!!!! She is now 6. My fiancee also calls me Lil Momma!! And he never once told her anything, NOR defended me!!!!! Yet I'm the one who washes his clothes, irons them for school, washes his towel for school on Mondays, I make sure his school notes are in his booksack, and when my fiance is offshore i bring his son to and from school and anywhere he needs. Anytime i am in town, i pick something up for his son whether he's with us or not. I see him as if he were my own son. But now, my fiance is letting his son and ex run our schedules and lives. I got mad this past weekend because his son was sick while he was offshore so he went back to his mom's until he felt better. Then that's when his ex and his son started saying WHEN he would come back, they made plans w/o talking to me because we have the kids on the same weekend and we have a break on the opposite weekends with just my fiance and I, well the plans made, totally screwed up our plans for the weekend and he didn't even ask me. Then as we're getting into it and arguing, he acts like he doesnt care and always regrets the ugly things he says later but says extremeley hurtful things while the argument is going on. As I was talking last night I said "you say you love me, you love me, you love me, and i'm the one you want to be with," and right before i could finish my statement he said "I guess i don't" i was stumped and didn't know what to say, yet he still came to bed and held me during the night, and this morning brought his son to school (my daughter is sick) and came to give me a kiss and told me he loved me. He went to the office to turn in his time and he came to kiss me and tell me he loved me again, but this time i didn't say it back. AM I WRONG?