Is he cheating or just has a close female friend?

My boyfriend and I have been together for 4 years, 2/1/2 of which we have lived together. I have always trusted him until this past Jan. when he lied about being invited to brunch by a male co-worker, when instead it was a female co-worker who is divorced. I have been suspicious of him since then even though he is accountable for his time out of work( I work in a retail enviroment, and he works Mon-Fri days). I started checking phone logs and see that the frequency of calls has increased over recent months...last month there were 14 calls and 11 texts. My previous boyfriend cheated on me several yrs. ago and that has definitely affected my thoughts about this. This is the only topic we ever argue about because I know the call history. I have told him that he makes it seem like he's hiding something because he always gets very defensive whenever I bring up my concerns. He claims they are JUST friends, but my gut feeling is either spot on that there is more or I am looking to make myself miserable. Next year he knows there is a very good chance that he will have to move because his current job is ending. We have talked about possible relocations as a couple. I own the house we live in and moving would mean selling it. I want to be with him because I do love him. But I just can't seem to have a constuctive conversation about the topic of this female co-worker without him getting irate. We have never discussed marriage, although his adult daughter just recently asked him about the"m" word. According to her, he said something about "a woman at work that was interested in him". When I asked him what one thing had to do with the other, he once again gave me a defensive "no answer" answer. If I didn't check phone logs I probably wouldn't think about this whole thing so much, but I got burned once....What do you think and what would you do?