Should I continue a relationship that seems to be going nowhere?

I have been dating a divorced man of 3. He has young children and an older 18 year old that lives with him. We have a more than 10 year age difference. When we met four years ago I made it clear I was passed playing mommy or any of that other kid stuff. We lived together (he in my home for 18 months) his 18 year old made it impossible for me. She talked back was lazy, and anything I needed done had to be asked thru him. I asked him to move out.
We continue to see each other. I thought once the 18 year old finished school our lives would change. She would hopefully go on to college. Instead she is attending communtiy college and living off and with Dad.
Our hopes of living together are up in the air.
Recently he reconsidered moving back in with me irregardless of her thoughts on it. Then he unexpectedly changed his mind. So we are back to my house his house.
He says I have to prove my self to him. That I can accept his family and be a part of all of it.
I am divorced, kids grown and gone. I feel he is just stalling and at my age I am looking for a mate. Someone to share time and life with. He has his kids every weekend, although most times I am invited along. I usually feel like a 3rd wheel. Recently the 5 year old made it clear he doesnt like me along.
Even after 4 yrs I love him tremendously and understand the kid thing. I just feel he expects too much of me.
Over the past 4 months he has become distant and looks for excuses not to spend time with me. It seems its usually on his terms.
What should I do. I feel weak, insecure and lonely.