Within a landscape, multiple habitats exist for soil microbial communities. But how these habitats shape community composition requires an understanding of the way in which microbial diversity is impacted across a broad range of spatial scales. Mountain ecosystems are excellent systems to study microbial communities, because a multitude of climate and soil variables change within a relatively small distance. We investigated microbial community structure in bulk and rhizosphere soils beneath three plant species, Vaccinium myrtillus, Juniperus communis and Picea abies, that structure local plant communities along an elevation gradient in the French Alps. We examined the impact that climate, soil properties, plant diversity and plant root chemical and morphological traits had on microbial αand β-diversities. The most abundant bacterial phyla detected in both bulk and rhizosphere soils were Proteobacteria, Actinobacteria, Acidobacteria and Verrucomicrobia. Along the elevation gradient, bacterial phyla did not display a clear distribution pattern between bulk and rhizosphere soils. For fungi, dominant phyla were Ascomycota and Basidiomycota, and contrasting distribution patterns were found between bulk and rhizosphere soils. Overall, bacterial and fungal α-diversity responded differently to elevation as well to soil compartments (bulk versus rhizosphere soil), revealing no significant patterns in bulk soil beneath any of the structuring plant species, but increasing in the rhizosphere compartment of P. abies just below the treeline. Changes in bacterial β-diversity with elevation were related mostly to soil physical and chemical properties. Bacterial and fungal α-diversity in rhizosphere communities were more related to plant species identity, vegetation diversity and belowground plant traits compared to soil properties, whilst the opposite was found for bulk soil. Our results highlight that environmental changes at the landscape scale (e.g. associated to elevation, soil properties or climate), impact significantly soil microbial communities, but vegetation refines communities at a local scale via the rhizosphere niche.