He's At It Again!

I have been married to my husband Brian for 27 years. He's always been busy and helpful in the neighbourhood, however, recently Brian has been behaving in a very stange fashion. My neighbour Maxine was sitting at the table when I got home last Wednesday, the front door was slightly ajar so I let myself in quietly as there was no latch sound. Maxine was sitting with her back towards the hall, she was moaning slightly, at first I thought she was stroking the cat, but on further inspection I noticed Brian's head bobbling just in front of her at waist level. At this point I tripped over the cat (the one I thought Maxine was stroking) and both Brian and Maxine jumped to their feet. Maxine ran for the door and Brian has refused to talk to me about it, simply saying 'it's all in your head, don't you trust me?'. Since then Brian has been sleeping on the sofa and each time I broach the subject he refuses to talk about it. The trouble is my mind has been full of past incidents that at the time I brushed aside any doubt about his intentions, but the more I think about it I am starting to think he has been playing away. For instance, when he put aside one evening a week for three monhts in order to errect a shelf in Pauline's kitchen, and the other time where he gave Mrs Higgins a lift to the WI every Tuesday evening between March 2002 and Apri 2004. After 27 years of marriage I want to trust him but my instinct tells me that I shouldn't and I should kick him him out. Please advise me what I should do, I feel in total despair. Thank you
Ange