If I were you I would walk in rocking out (Means wearing it proudly) a sweatervest,If someone thinks that's "over the top" Don't worry about it, I would wear a sweatervest to school,Someone would ask me, Are you going to church tonight? I would reply no I just like wearing it.
I work in high tech and it is the same here - at least at the first 2-3 levels in the organization - but when we have to go to customers more often the dress goes up too.
I said screw it and wear a sport jacket everyday. A suit is too much I agree and I am considering seeing what the reaction to knit ties are when it starts getting warmer and I shed the jacket.
I am not a sweater fan as I need to adjust to all of the temperature ranges here so a jacket work better. Everyone is used to it now and I get very few comments any more. It also feels great to be the rebel by not wearing the uniform.