Is it wrong for me to go to a party my girlfriend wasn't invited to?

I have a circle of friends who aren't really fond of my girlfriend for differing religious opinions. They are my friends though, and they are having a party in which they said I couldn't bring her. The party also happens to land on a night I had plans to take my girlfriend out. Is it wrong for me to cancel our plans and go to the party? If it is wrong, is it because she is not invited, or is because I'm cancelling plans?