When people have "made-up" feelings for others outside the relationship, do you stay together or keep trying?

I've been with a guy for over three years now. Firstly, we do live apart. We get to see each other twice a year. Early last year we broke up because he met a woman and said he'd fallen for her, only to tell me soon after that he'd misunderstood a simple crush. These crushes happen because he needs to feel love, he says. Last December another woman came into his life and the same thing happened. Only this time I am unsure as to whether to take him back or not, to give the relationship another chance.