Within-population variation in individual tree growth and response to climate has an impact on forest dynamics, resilience and adaptation to environmental change. Combining dendrochronological analyses with a process-based ecophysiological model simulating drought stress at the stand scale, we studied the phenotypic variation of two growth-related traits within 22 pure stands of five contrasted tree species sampled in the RENECOFOR network over a wide range of ecological conditions. First, we computed the annual stress level from soil, climate and stand inventory data. Second, we computed individual sensitivity as the quantitative growth response to drought stress level and individual vigor as the capacity to grow in favorable years relative to an average stand-level growth model. We analyzed within-population variation and covariation of individual vigor and sensitivity, their temporal changes during stand development, as well as the effect of environmental conditions on population-level means, variances and correlation. Our results show that within-population variances in sensitivity and vigor exceed the between-population variances for all species. The populations located in more stressful environments, i.e., low summer precipitation and extractable soil water, showed lower mean and variance of sensitivity, suggesting possible multiscale adaptation at the population level and within populations. None of the environmental factors considered had an effect on the average population vigor or on the within-population variance of vigor. We found a general positive correlation between individual growth sensitivity and vigor in 17 out of 22 populations, potentially revealing a growth performance trade-off. The correlation was more pronounced in low extractable soil water environments, which may be related to a need for stressful conditions to reveal the trade-off or be the consequence of adaptive processes, i.e., acclimation and selection. If high within-population stand phenotypic variation in growth traits contributes to the resilience and adaptive capacity of forests to climate change, a trade-off could represent a constraint on selection. We provide genetic and environmental arguments supporting the hypothesis of a trade-off, then we highlight the importance of integrating it into the management process, especially during selective thinning, to avoid indirectly increasing population sensitivity by selecting the most vigorous trees.