Background: Plant phenological traits such as the timing of budburst or flowering can evolve on ecological timescales through response to fecundity and viability selection. However, interference with sexual selection may arise from assortative mating. This study aims to investigate how these three components of selection on spring phenology may combine in European beech populations in contrasting environments (high versus low altitude). Methods: we monitored the timing of budburst (TBB) in 339 adult beech trees and estimated their fecundity using spatially explicit mating models. Fecundity selection was infered by regressing fecundities on TBB, while sexual selection was inferred by regressing fecundities on mating opportunities (i.e., TBB mismatch). The correlation between mates for flowering time (i.e., assortative mating) was estimated based on paternity analyses. Morever, TBB and growth were surveyed in 3261 seedlings from 40 families grown planted in a common garden, and viability selection was inferred by regressing growth on TBB. Results: Overall, directional fecundity selection on female fitness favored trees with earlier TBB. Sexual selection acted only on male fitness through assortative mating favoring trees with mean TBB value (stabilizing selection). In the common garden, early budburst was associated with higher seedling growth. The respective intensities of directional and stabilizing selection varied with the environment: at low altitude, directional selection for earlier phenology was modulated by strong assortative mating and by an interaction effect between TBB an size on female fecundity, whereas at high altitude, directional selection for earlier phenology was reinforced by selection through male fecundity. Discussion: This study showed that selection through female fecundity and seedlings growth predominantly selected for earlier TBB, while sexual selection on male fitness through assortative mating modulated this trend. This interplay between fecundity and sexual selection calls for an integrative approach to predict the evolution of spring phenology under a changing climate.