cashmere sweaters are expensive, look for merino wool or alpaca, they will keep you just as warm and are easier to find for men.
Also, look at all the labels for your clothing, anything that says "cotton" should be removed and replaced with merino wool, or a synthetic material. (denim=cotton 0h, and don't forget to replace your underwear with a synthetic as it's most likely cotton as well.)
Cotton is a terrible fabric for travelling. When it gets wet, it stays wet. It attracts and holds onto stains and funky smells. It's difficult to wash & dry unless you have access and time to run it through a machine.
Synthetics (and merino wool) can be washed in the sink of your hotel (or campground/river whatever) and will be dry the next morning.
Look for
ExOfficio brand as a starting point.