Interspecific hybrids of Populus species are known for their superior growth. In this study, we examined the effect of the genetic background and contrasting environmental conditions on growth and searched for quantitative trait loci (QTL) affecting growth traits. To this end, two hybrid poplar families resulting from controlled crosses, Populus deltoides ‘S9-2’×P. nigra ‘Ghoy’ (D×N, 180 F1) and P. deltoides ‘S9-2’×P. trichocarpa ‘V24’ (D× T, 182 F1), were grown at two contrasting sites, Northern Italy and Central France. At the end of the second growing season, tree dimensions (stem height, circumference, and volume) were assessed. The performances of both families significantly differed within and between sites. Tree volume was significantly larger at the Italian site as compared to the French site. Genotype by environment interactions were significant but low for both families and for all growth traits. Tight correlations among the individual growth traits indicated that there may be a common genetic mechanism with pleiotropic effects on these growth traits. In line with previous studies, linkage groups I, VII, IX, X, XVI, XVII, and XIX appeared to have genomic regions with the largest effects on growth traits. This study revealed that (1) both families have high potential for selection of superior poplar hybrids due to the pronounced heterosis (hybrid vigor) and the large genetic variability in terms of growth and (2) the choice of site is crucial for poplar cultivation.