In most polygynous vertebrates, males must allocate energy to growing secondary sexual characteristics, such as ornaments or weapons, that they require to attract and defend potential mates, impacting body condition and potentially entailing fitness costs. We investigated sex differences in over winter body mass change across five intensively monitored populations of roe deer (Capreolus capreolus) with markedly contrasting environmental conditions. At winter onset, males weighed, on average, 8.4% (from 4.7% in the most northerly population to 11.6% in the most southerly one) more than females. However, across all populations, males fared worse over the winter than females, losing more (Sweden) or gaining less (France) mass, so that sexual mass dimorphism was virtually absent prior to the onset of spring. Our findings reveal that the direction of over-winter change in mass of roe deer depends on winter severity, but that males are consistently more sensitive to this environmental constraint than females. As a result of this sex-specific change in body mass, sexual mass dimorphism is lowest at the onset of the territorial season. We suggest that allocation to antler growth and territory establishment drives this pattern, providing a likely explanation to account for the lower rates of male adult survival that are consistently reported in this weakly dimorphic species.