We're not going to get pregnant if we don't have sex during my "fertile week." He doesn't get my frustration?

We have been trying to get pregnant for just a couple of months, but we have just been shooting in the dark (pardon the expression) until this month. After talking to my doctor about my fertility chances at age 36, we decided to be more proactive about that fertility week when I am most likely to get pregnant.... great idea until our window of 5 days is down to 2 because we just can't get it together. He fell asleep early the first day. Second day I had to work late and didn't feel like initiating (not that I turned him down! I just didn't feel like being the one to start the process). Today I got up early so we could "make a baby" before he went to work, but he turned me down because we "didn't have enough time" (which was apparently MY fault). I'm really frustrated because he doesn't seem to understand. There is a STRONG chance that I will have difficulty getting pregnant. There is 100% chance of failure if we aren't even having sex!!! How can I get him to understand my frustration? It feels like I'm the only one trying here which makes me think he's not being completely honest about wanting a baby...