Surface runoff is the source of severe hazards such as floods, erosion or mudflows. Railways are particularly vulnerable to runoff as they can disturb natural surface water flow path. This can generate accelerated deterioration of the infrastructure, or the destruction of railway elements, such as the roadbed or the embankments. The IRIP model 'Indicator of Intense Pluvial Runoff' (French acronym, Lagadec et al., 2016) allows mapping the susceptibility to surface runoff through three maps of susceptibility to generation, transfer and accumulation of intense runoff. The IRIP model aims to be simple and reliable, in order to be used on all types of lands, without calibration stage. A proper evaluation of the relevance of the produced maps is difficult due to the lack of direct measures of runoff. However, evaluation is a crucial stage to estimate the value and the reliability of the model outputs. These information are essentiel to pursue model developments especially in the perspective of an operational use. In this study, we present an evaluation method based on proxy data such as impacts of surface runoff on railways. Difficulties lie in the fact that data to be compared differ in terms of form and content (susceptibility maps versus localized impacts). Moreover, the infrastructure vulnerability must be characterized and taken in the evaluation process for the latter to be effective. The proposed evaluation method takes into account the vulnerability of the infrastructure and mitigation measures before computing contingency tables and verification indicators. The method is applied on 80 km of railways in northern France, over which an historical data base of impacts of surface runoff, dating back to the beginning of the 20th century, has been collected. On this study area and with the proposed evaluation method, the IRIP model shows promising results with a probabilily of detection of about 90% and a false alarm ration of about 20%. The evaluation method also allowed assessing various options for improving the IRIP model. The study contributes to improved knowledge about surface runoff and provides clues for improving the risk management of surface runoff hazard in the railway context. It confirms good prediction results alredy obtained with the IRIP model in other contexts of application.