Inserting legumes in low-input innovative cropping systems can represent a good strategy to reduce current N fertilizer dependency while enhancing ecosystem services. However, although the impact of the use of legumes as cover crops has been broadly studied, very little is known about the effects of grain legume-based rotations on soil organic carbon (SOC) and nitrogen (SON). A cropping system experiment with three 3-year rotations with different levels of inclusion of grain legumes: GL0, GL1 and GL2 (none, one, and two grain legumes, respectively), with (CC) or without (BF, bare fallow) cover crops was established in SW France (Auzeville) under temperate climate. Durum wheat was present in all the rotations to act as an indicator of their performance. Soil organic C and SON were quantified before the beginning of the experiment and after 3 and 6 years (i.e., after one and two complete 3-yr rotations). Aboveground C and N inputs to the soil, and C and N harvest indexes and grain yield of the cash crops were also measured. Inserting grain legumes in the rotations significantly affected the amount of C and N inputs and consequently SOC and SON. After two cycles of the 3-yr rotation, the GL1 and GL2 treatments showed a greater decrease in SOC and SON when compared to GL0. However, the inclusion of cover crops in the rotations led to mitigate this loss. Durum wheat produced significantly greater grain yields in GL1 when compared to GL0, while GL2 presented intermediate values. In turn, the incorporation of cover crops did not reduce C and N harvest indexes or the grain yield of the different cash crops. We concluded that, in such conventionally-tilled grain legume-based rotations, the use of cover crops was efficient to mitigate SOC and SON losses and then increase N use efficiency at the cropping system level without reducing productivity.