what should i think?

My boyfriend has pretty much an ex girl friend they never techinically dated but they were together on and off for a year or two. A few weeks ago they talked on the phone all night for about 4 hours and he texts her and whenever i say anything bad about her when i get mad he gets mad i say something about her and sticks up for her.. should i be worried about this or am i just being insecure?