Exactly.
I think most non smokers who say it doesnt bother them are just trying to not rock the boat. Usually (in my opinion) because when the average smoker hears someone criticise the habit, they have an enormous over reaction, and get near violently angry and will state that such an opinion is in violation of their rights.
Upon hearing the rebuttle "well what about my right to not have to inhale the toxic fumes from your (tobbacco product)?" you will usually get a response like "well you have a right to go somewhere else where you won‘t smell it", or some other extremely stubborn statement.
People get stubborn when they defend an illogical point of view, like a child demanding to know why they can‘t eat those cookies a 1/2 hour before dinner, they don‘t care that there is a valid reason, they just want their way. So they throw a fit.
Its hard to be reasonable, when you know even before you begin an argument, that you are clearly in the wrong. And I think the vast majority of smokers know that it is a harmful habit, and the wiser part of themselves wants to quit.
This is why the "Quit smoking" industry is laughing and rolling around in their money. They know the quickest fix is to just make a choice to quit, but they also know that if people feel as though they are making even a tiny effort (chewing nicotine gum, nicotine patches) they will feel good, and usually reward themselves with a few puffs, thereby perpetuating the cycle.