If I were you I would walk in rocking out (Means wearing it proudly) a sweatervest,If someone thinks that's "over the top" Don't worry about it, I would wear a sweatervest to school,Someone would ask me, Are you going to church tonight? I would reply no I just like wearing it.
I work in high tech and it is the same here - at least at the first 2-3 levels in the organization - but when we have to go to customers more often the dress goes up too.
I said screw it and wear a sport jacket everyday. A suit is too much I agree and I am considering seeing what the reaction to knit ties are when it starts getting warmer and I shed the jacket.
I am not a sweater fan as I need to adjust to all of the temperature ranges here so a jacket work better. Everyone is used to it now and I get very few comments any more. It also feels great to be the rebel by not wearing the uniform.
Get a good blue blazer with brass buttons. It's always casual, and you can wear it with jeans but also with other pants. Get one that is classic cut, and it will be in style forever.
Of course, white oxford always works.
I am too hard on shoes to do this, but the men I see who are really pulled together in casual clothes always have great classic shoes...