Should I be upset by my husband getting friendly with gym bunnies?

My husband plays racquetball in a weekday evening league. We have a son who is 8. I stay home with him since the start time for my husband's games are as late as 9:30pm and it is usually too late for our son to be up. This past evening my family and some friends went to the club for swimming and so one. While were seated in the food court area a woman walked past and stroked my husbands neck and gave him a very warm hello. I was seated behind a pole and wasn't visible to this woman. A male friend of ours who was seated at the table made a remark similar to…”Wow!...Who is that? She was pretty friendly! a stroke along the neck?”. He was clearly surprised by the intimacy of the gesture as was I. My husband said, “I don't even know her” and turned quite red. He later said they played couples racquetball together in a “pick-up” match one night after a league game. The intimacy of the gesture indicated to me that the couples match was probobally flirtatious. That combined with the fact that he never mentioned the the match is making me feel betrayed. He says he didn't do anything wrong.