Background: Understanding the genetic and environmental mechanisms governing variation in morphology or phenology in wild populations is currently an important challenge. While there is a general consensus that selection is stronger under stressful conditions, it remains unclear whether the evolutionary potential of traits should increase or decrease with increasingly stressful conditions. Here, we investigate how contrasting environmental conditions during growth may affect the maternal and genetic components of body mass in roe deer, the most abundant and widespread wild ungulate in Western Europe. Body mass is a key life history trait that strongly influences both survival and reproductive performance in large herbivores. We used pedigrees and animal models to determine the variance components of juvenile and adult winter body mass in two populations experiencing contrasting early-life conditions. Results: Our analyses showed that roe deer at Chize, where habitat was poor and unpredictable, exhibited very low genetic variance in juvenile body mass. Instead, variance in mass was mainly driven by among-cohort differences in early-life conditions and maternal environment. In contrast, roe deer at Bogesund, where resource availability during the critical period of fawn rearing was higher, displayed a substantial level of genetic variance in body mass. We discuss the potential role of past demography and viability selection on fawn body mass on the erosion of genetic variance in the poor habitat. Conclusions: Our study highlights the importance of accounting for both spatial (i.e. between-population variation) and temporal (i.e. cohort variation) heterogeneity in environmental conditions, especially in early life, to understand the potential for adaptive responses of wild populations to selection.