Any of those would work, and I would also suggest considering a tan colored sport coat as well. Your suits (black and pinstripe) are fairly dressy, so I think something toward the other end of the spectrum (light grey or tan) might best fill a gap in your wardrobe.
Or tweed. Possibly houndstooth. In spite of a large selection that's what I usually take out of the closet.
+1 on the houndstooth and tweed .
Blazers have been taken over by the customer service people, who wear them like a uniform. I would suggest a light brown Spring/Summer weight sport coat. The sales woman I bought my brown from said that browns have color tones, and I have one that is kind of red, and goes with just about everything I wear.
Navy hopsack will be your most versatile, if you get a lightweight one it can be worn in summer by itself or winter with a thicker shirt and a cardigan or sweater vest underneath. It will go with jeans, chinos or grey flannels, and that gets you from street casual right up to just shy of a suit in terms of formality. Navy also goes with pretty much everything.
Don't cheap out. If you're going to be wearing this a lot you need to get a good fabric in a garment that is assembled well, and you'll want it MTM, bespoke, or at the very least tailored well to fit you. if you need suggestions on brands to get, I'm more than happy to answer questions via PM.
Tweed is more casual, more fun, and imho more comfortable, but you're not going to be able to hit it if the mercury hits the 60 degree mark. Save tweeds for later.
For a more casual option, what about something like this: