Arbuscular mycorrhizal fungi (AMF) play a central role in rhizosphere functioning as they interact with both plants and soil microbial communities. The conditions in which AMF modify plant physiology and microbial communities in the rhizosphere are still poorly understood. In the present study, four different plant species, (clover, alfalfa, ryegrass, tall fescue) were cultivated in either sterilized (gamma ray) or non-sterilized soil and either inoculated with a commercial AMF (Glomus LPA Val 1.) or not. After 20 weeks of cultivation, the mycorrhizal rate and shoot and root biomasses were measured. The abundance and composition of bacteria, archaea, and fungi were analyzed, respectively, by quantitative PCR (qPCR) and fingerprinting techniques. Whilst sterilization did not change the AMF capacity to modify plant biomass, significant changes in microbial communities were observed, depending on the taxon and the associated plant. AMF inoculation decreases both bacterial and archaeal abundance and diversity, with a greatest extent in sterilized samples. These results also show that AMF exert different selections on soil microbial communities according to the plant species they are associated with. This study suggests that the initial abundance and diversity of rhizosphere microbial communities should be considered when introducing AMF to cultures.