What do I do when he only wants to have sex in the middle of the night?

I feel like whenever I try to initiate sex my boyfriend rejects me. Either he is tired or he just jokingly says no so I stop trying. When he does want to have sex usually we are already asleep for an hour or two, and I never say no because it feels like that is the only time we are going to do it. He says it's because sometimes we aren't in the mood at the same time. Isn't part of the fun getting each other in the mood? And when he wakes me up in the middle of the night he doesn't take the time to get me in the mood either.

I don't know how to approach this. I love that he is spontaneous like that, but I am starting to not want to even attempt to initiate because of the potential rejection.