The rich were able to have their shirts custom-made by professional shirtmakers. Those who weren't so well-off bought shirts that were only measured in neck-circumfrence. Everything else (sleeve-lengths included) were extra large to fit as many people as possible. So if all you could afford were the cheap shirts, then yes, you needed sleeve-garters.
They only look 'dumb' if you don't know how to pull them off. Making something look good is a matter of having the balls to wear it with confidence and style.