Soil water erosion and shallow landslides depend on aggregate stability and soil shear strength. We investigated the effect of vegetation on both soil aggregate stability and shear strength (through direct shear tests) in former croplands converted to vegetated erosion protection areas within the context of China's sloping land conversion programme. Four treatments were analysed in plots comprised of (i) 4 year old crop trees, Vernicia fordii, where understory vegetation was removed; (ii) V. fordii and the dominant understory species Artemisia codonocephala; (iii) only A. codonocephala and (iv) no vegetation. Soil samples were taken at depths of 0–5 cm and 45–50 cm. Root length density (RLD) in five diameter classes was measured, soil organic carbon (SOC), hot water extractable carbon (HWEC), texture and Fe and Al oxides were also measured. We found that mean weight diameter after slow wetting (MWDSW) in the A-horizon, was significantly greater (0.94–1.01 mm) when A. codonocephala was present compared to plots without A. codonocephala (0.57–0.59 mm). SOC and RLD in the smallest diameter class (< 0.5 mm), were the variables which best explained variability in MWDSW. A significant positive linear relationship existed between MWDSW and soil cohesion but not with internal angle of friction. As herbaceous vegetation was more efficient than trees in improving aggregate stability, this result suggests that the mechanisms involved include modifications of the cohesive forces between soil particles adjacent to plant roots and located in the enriched in SOC rhizosphere, thus also affecting shear strength of the corresponding soil volume. Thus, vegetation stabilised soil under different hierarchical levels of aggregate organisation, i.e. intra- and inter-aggregate. Our results have implications for the efficacy of techniques used in land conversion programs dedicated to control of soil erosion and shallow landslides. We suggest that mixtures of different plant functional types would improve soil conservation on slopes, by reducing both surface water erosion and shallow substrate mass movement. Planting trees for cropping or logging, and removing understory vegetation is most likely detrimental to soil conservation.