My roommate brother has been using my bar soap in the shower. I'm now out of soap and he won't let me use his

My brother and I are college roommates along with 3 other guys. Everyone has their own soap in the shower except my brother. He has been using my soap thinking that I don't know, but I do know. So now I am out of soap. However, he just got a bottle of body wash. I told him that I wanna use it until I get a chance to get my own soap. He refuses to let me use his body wash. Shouldn't I be able to use his soap since he has been using my soap for over a month now??