Am I taking this to serious?

I have been with this man foor 25 years we have two grown daughters, we were married last year, in a beutiful cremony, recently he has started talking to his ex wife, she lives in Kentucky, they have been divorced for 30 years, but she has a lot of problems with her life, they talk on the phone, and when he goes to Kentucky, to see family she happens to see him, the last time he was there he came home with two letters she had given him, which stated she still loved him, and always had, the reason they divorced is she had an affair with the best man at the wedding, This really hurts me, because he lies about it I have to find out from his sisters the truth about everything that is going on, in return he gets mad because I will not put it to rest. he says they have issues they need to work out, now we have not spoken to each other in 3 weeks, but we still are living in the same house, please help me. what do I do? am I wrong?