In that vein, my father was a "collar and tie" man at work, he was open collar off the job. My grandfather was a rancher and commercial harvester. I've seen the photos of him driving a horse-drawn combine, cutting grain, and he's wearing a collar and tie.
I've seen any number of photos of work groups dating from roughly 1930 through 1950. The common denominator, collared shirts and neckties. With the exception of the fellows who unloaded the heavy stuff from the trucks, there's a collar and a necktie on all the guys in those photos. The biggest descent into casual is no suitcoat during the summer. And those fellows are still wearing neckties!
Those were the days, it is a shame we have gotten away from that as a society.
I wear scruff around the house (cargos, older shirts, t-shirts etc), but when I leave the house it is always in a fresh, clean, ironed set of smart clothes.
I don't see the point of going out like a scruff when I've put a lot of time, effort and cash into building a nice wardrobe over the years. Anyway, you never know who you will bump into on your travels, so it's always nice if you look good and feel that you are looking your best.