Do you think it is proper for Hispanics to speak Spanish in an office environment among English speakers?

I work in an office and there has been a big problem with Hispanics speaking Spanish in the office when there are several of us who do not speak it. I noticed that when a problem with or a disagreement, arise, they would speak Spanish amongst each other. We didn't know if they were talking about us or not but to me, it is extremely rude.