Should I waste anymore mental energy on this?

My colleague and I seem to have some chemistry going on but I cannot be sure - I've asked him to come to social outings a few times but he never turned up. He has asked me once but I got the feeling he was only inviting me because our cutesy intern was going.

He seems to be a bit of a flirt but when confronted he admits he's shy. I am at a loss as to what to make of this. Him being rather conservative in his own way doesn't open up all that well. I am beginning to feel rather fatigued having to work with him with all this ebbing in the background. Any suggestions?!
Re the backoff.. I was for two weeks treating him strictly as colleagues and ended up receiving some text about why I have been rather distant, and that we are "really good friends" .. this is all a bit high-schoolish to me that I dont know what to make of it!