An abundance of methods have been developed over the years to perform the frequency analysis (FA) of extreme environmental variable. Although numerous comparisons between these methods have been implemented, no general comparison framework has been agreed upon so far. The objectives of this presentation are: (i) to describe the foundation of a general framework devoted to the data-based comparison of FA implementations; (ii) to illustrate its application based on an extensive dataset of French gauging stations. The comparison framework is based on the following general principles: (i) emphasis is put on the predictive ability of competing FA implementations, rather than their sole descriptive ability measured by some goodness-of-fit criterion; (ii) predictive ability is quantified by means of reliability indices, describing the consistency between validation data (not used for calibration) and FA predictions; (iii) stability is also quantified, i.e. the ability of a FA implementation to yield similar estimates when calibration data change; (iv) the necessity to subject uncertainty estimates to the same scrutiny as point-estimates is recognized, and a practical approach based on the use of the predictive distribution is proposed for this purpose. This framework is then applied to a case study involving more than one thousand gauging stations in France, where several FA implementations are compared. These implementations correspond to the local, regional and local-regional estimation of Gumbel and Generalized Extreme Value distributions, completed with a “derived distribution” approach based on a rainfall simulator coupled with a rainfall-runoff model. Results show that reliability and stability indices are able to reveal marked difference between FA implementations. Moreover, the case study also confirms that using the predictive distribution to indirectly scrutinize uncertainty estimates is a viable approach, with distinct FA implementations showing marked differences in the reliability of their uncertainty estimates. The proposed comparison framework therefore constitutes a valuable tool to compare the predictive reliability of competing FA implementations, along with the reliability of their uncertainty estimates.