Should my husband end Friendship with Couple who were friends with he and his first wife?

My husband was married to his first wife for 8 years before he and I met. They were friends with another married couple for quite some time. When he divorced his wife and married me, this couple that he was friends with made it clear they did not like me. They would come to our house and speak to him and not me. They will always talk to him when they see him out in public and if I'm with him they give me a dirty look. They are kinda childish and the woman always gossips and causes problems. I think these people are immature and I told my husband out of respect for my feelings he should cut ties with them completely.