Surface runoff leads to severe hazards such as floods, erosion or mudflows. Railways are particularly vulnerable to runoff as they can disturb natural surface water pathways. The IRIP method 'Indicator of Intense Pluvial Runoff' (Lagadec et al., JoH, 2016) allows mapping the susceptibility to surface runoff through three maps of susceptibility to generation, transfer and accumulation of runoff. The IRIP method evaluation faces the lack of direct observation of runoff and must use proxy data. The paper presents such an evaluation method using impacts of surface runoff on railways. The method takes into account the differences in data (susceptibility maps versus localized impacts), as well as the vulnerability of the infrastructure and mitigation measures before computing contingency tables and verification indicators. The method is applied on 80 km of railways in northern France, over which a 100-year historical data base of impacts of surface runoff has been collected. The IRIP method shows promising results with a probabilily of detection of about 90% and a false alarm ratio of about 30%. New developments examine how information about rainfall hazard can be taken into account to define exposure and offer perspective to assess the impact of climate change on intense runoff risk.