Is our relationship really worth it?

So I met this guy about a year ago, and we dated for 4 months, then one day he said he didn't love me anymore, it was like he loved me as a friend. So we were separated for almost two months, and I couldn't stop thinking about him and I missed him everyday. Then he said he wanted to have another shot at our relationship, because he noticed he too had been missing me, and that he really cared about me. We dated for 2 weeks and then he said he felt like he didn't want to continue anymore, so he left me, again. Now I've been seeing him again after a short brake, and he said he has never loved anybody as much as me, and I really feel the same about him. The problem is he kind of neglects me from time to time, but its a little bit in his nature, and most of the time hes really sweet, but I feel like the neglecting has been increasing lately. We get into fights about him not giving me as much as attention as I would want from him, nothing totally out of line, I just want him to be interested in what I do, and ask me simple questions like "How was your day?" or "What were you planning to do today?" and talk with me in general. But we fight about some really stupid stuff at times, small things that we make serious, but they necessarily aren't serious at all. And he gets annoyed very easily, so I always have to watch what I say. Sometimes it feels like he doesn't appreciate me enough, though at this time we're not officially in a relationship, but we're not broken up either. And at least for him that may be a factor which affects his behavior towards me. I also don't like the way he talks to his girl friends, its a little bit too intimate for me at times, and I understand that people are different and have different kinds of relationships with people, but I feel like he likes talking to them more than me. Despite all this, I think he has feelings for me, why else would he want me back? And I really want to be in his life, so that I'm not a separate event in his calendar, but a part of his everyday life, if you know what I mean. But I don't know how to tell that to him in a way he won't get all uneasy, because one of the reasons he left me before, was that he felt I "demanded more time than he had to give" (he's busy, he plays hockey, and is serious about it) Just a while ago we again fought over meaningless things, and the fact that he wasn't openly interested in my doings as I was his, and he really got annoyed about the fact I thought hes not interested (yes who wouldn't, I know) but I have a bit of temper, and tend to say things I regret MAJORLY after I say them. So the question is should we keep trying?