Should I say something in my defense?

I had been dating my boyfriend for about six months, then I moved in with him and his roommate, us sharing a room. Right after I moved in my boyfriend told his dad. His dad got upset (him being very religious) he was upset that I moved in and we aren't married. Now I felt kinda bad b/c my bf and his dad are super close, I talked to my bf and he said he still stands by our decision to move in together, now my bf is 29 and Im 22, and we are very mature, thoughtful people, so I think our decision was correct. Now recently my bfs grandmother came in from Ireland to visit and she took us all out to dinner, and his dad didnt come, and I wanted to try and settle things with his dad b/c I hate drama and tension, so I tried to see if his dad wanted to come over for dinner and maybe a card game of some kind, and he told my bf that he wouldnt set foot in our apartment b/c I lived there. Should I say something, should I let it go? what do you think