Why do some non smokers go out of their way to harass smokers?

I smoke, and have been for the last 15 (+ or - 2 years), yeah can't remember that far back lol. I try to consider myself a consicous smoker, meaning I am aware of what's around me and who so that if someone who comes near me I will instinctivley move away a few steps as well as try to watch where smoke goes.
I should say that I smoke in designated areas on my college campus, and if I'm off campus I generally smoke when no one else is around.

However, recently I have been receiving comments from people about how rude I am towards them by smoking. These people aren't even around me when I light up, they come over to where I am and then make the commment. They ignore me when I ask them if putting out my cigarette would appease them.
I get the "evil eye", I get treated like some murderous fool, and I made to feel like an outcast of society. I don't even get the choice to use my own money to build shelters to code to provide more convenient places on our campus.
I just don't understand how people will walk towards a group of smokers who are not on the main walkway and then complain about how bad the smoke is or how we "Blew our smoke in their face" (actual quote from a complaint).

I know that smoking is unhealthy, and that it doesn't just affect me. However, can't there be some comprimise? If we smokers are the only ones in a particular area, can't we have it a Smoker's Only area?
I mean how many non-smokers go out of their way to consider other people during their day?

Updates
So I see alot of people here are saying that smokers stink or that I should quit smoking. I'd like to make this a double question by asking how many people say something to someone who has overally bad body odor?
I believe that society, egged on by a virulant media campaign, has targeted anyone with a cigarette, pipe, or cigar in their hands as a "bad person". Perhaps that's the answer to my orginal question?---
____________________________________________________________

Update based on the selected best answer:

____________________________________________________________

I guess you're right, but still, if I'm off by myself minding my own business and they approach, then they are interfering with my lifestyle. It would be no different if I went up to someone in a bar and told them that drinking that shot will endanger his life and those around him. Yet how many times do we see people who consume alcoholic beverages persecuted as hard, non criminally that is?
Everything can be debated, regardless of the facts and who might be wrong or right. I don't think that just because the facts support the argument gives a person the right to come up to me when I'm alone and justify their righteousness.
Now I understand you're point that if i'm in a crowd with other smokers who are less than courteous and lumping me in with that group, and in that case, i concede and will take any said reprisal. Yet there again, if we smokers congregate outside and from the main thoroughfair, we will still see people come up to us or make exagerrated motions with waiving of arms or pinching of noses when they could have simply walked on the other side of the walkway.
I think, based on the responses I've seen here, that there really is no answer save for the fact that people will follow the main stream to avoid conflict. I'm not saying that smoking is good, but it would seem that the main stream media fueled by society's views would make us out to be villains worthy of crucifiction.
So i'll concede and close this question, though I encourage all of you to consider what would it be like if society deemed something that you eat or drink or even do as bad for you and those around you. How would you feel if you tried to be considerate and kept that action to yourself only to be sought out and ridiculed?