I was going to say this. Try calling a sergeant "sir" and not getting yelled at.
I do chuckle a bit when a civilian sees me in uniform and calls me, a private, "sir".
I've always called my teachers and professors "sir" or "miss". Or, for my French courses: "monsieur" and "madame". They've never asked me to say otherwise, and I guess it's because they know that I say it out of respect.
"Ma'am" isn't very common around my neck of the continent. If it was, it has fallen to the wayside, replaced by "miss".
Rereading To Kill a Mockingbird, I find it interesting how Scout and Jem call their father "sir".
Being serious...I love being called Mr. my students call me Mr. Sir does have its time and its place. I still get called young man..because I'm only 26...usually from my barber..who is much older than me which is fine.
When It comes to addressing women I err to miss..I feel that some might take as a compliment (the same as giving them the compliment that they look young). When I hear Ma'am or Mrs. I usually think of a lady who is much older than I and usually married.
Way I operate, if you've got more than a decade on me, and I'm not calling you sir or ma'am, we're either real close or we have issues that either need to be or are being resolved.
I have no problem being called sir myself. In fact, I rather like it. Much more than the diminutive "bud" or "bro" that seems to be in fashion. I'm a friendly guy, but over familiarity is not one of the better inventions of modern society.