Many conceptual rainfall-runoff models have experienced difficulties to simulate annual or longer-term volumes in various validation experiments (e.g. Coron et al., 2014). The scope of this study is to investigate the role of temporal variations of the runoff ratio (i.e. the ratio of discharge to precipitation) in these drastic losses of performance. To this end, a systematic procedure of split-sample tests (Klemeš, 1986) has been set up with several lumped hydrological models on a set of 630 French and Australian catchments to assess a statistically-robust dependence of models errors to changes in the average runoff ratio. We find that the runoff ratio changes between calibration and evaluation periods significantly affect models biases. An increase (respectively a decrease) in the average runoff ratio indeed results in model underestimation (respectively overestimation) of the average river flow in the evaluation period. Besides, the annual changes in runoff ratio are negatively correlated to annual biases for a wide range of catchments in our dataset. Rainfall-runoff conceptual models are unable in most cases to reproduce the variations in catchments behavior, and thus they stick to the average behavior they have learned during calibration.