Dinner at his folks- its too much

Recently married. He moved in with me a year before the wedding and we were living with my parents while we got our finances in order. Every sunday we'd go to his parents house for dinner (to spend time and hang out).

Wedding comes and goes- his parents choose not to participate as much as i'd wished. The mother did not want to dance with the son, they just sat at their table in did not get up really, they did not want to be involved in the planning, they did not want to help my mom plan the shower, they did not want to help pay for anything, etc. Fine. Whatever. We paid for a lot of the wedding and my family went above and beyond helping us take care of the rest.

Anyways, this really hurt me. I feel like they dont like me since they made no effort (i'm not upset about them not giving us a gift, or money or anything...i'd be happy with them giving their hand to help decorate or SOMETHING).

Now we are at our own place. I dont feel comfortable going to dinner there every week. I'd like to at least switch it up and see both parents (every other weekend). (In a dream world i'd love to have one weekend at his, one weekend at mine and one weekend alone together- dont think that will happen tho).

Brought it up last night and he all but freaked out.

I got him calm enough to realize its only fair if we switch it up but something inside me tells me he's not happy.

Whats the best way to bring it up again without him getting so mad? Am I wrong for not wanting to go all the time?