Is it unfair to not allow my boyfriend's brother (he's been convicted of smuggling drugs) to stay at our house

I live with my boyfriend in his house. We have lived together for almost a year. He has a brother that has been convicted of smuggling drugs (several years ago) to a foriegn country and was in prison abroad. The brother has been in the states for some time now and his record seems to be clean, althought he has never expressed remorse for what happened. One other fact - a third brother worked for the airlines for several years and lost his job because of the drug smuggling.

The brothers would like to visit, for a week over the holidays and I do not trust them in the house. I feel as though my boyfriend and I have worked hard to establish our careers and don't want to risk anything for someone who takes a chance smuggling drugs. I feel my boyfriend can invite any of his family members to the house, but if he has so little regard for our well being, then I will move out.

Any comments?