How do you tell a guy you work with you don't wanna hang out outside of work without making it awkward?

The guy that works in the cubicle next to me is ok...at work. I would never be caught dead hanging out with him outside of work and he's always asking about happy hour. I always make excuses but I would hang out with other people there. Just not with him.

How do I go about telling him this without it making for an awkward work environment? we just differ on too many things and I find myself just agreeing to end conversions.