The pending ban on glyphosate raises the general issue of long-term usability of chemical herbicides and pesticides. Torn between legitimate economic and ecological concerns, EU lawmakers and agri-food stakeholders need to rely on alternatives to chemicals, which entails a strong support to the development of new solutions. Autonomous weed killing solutions are among these concrete solutions, considering that navigation, vision and decision-making technologies have reached a level of mastery that allows considering realistically the deployment of such devices. However, the technical feasibility and overall usability of autonomous agricultural machines need yet to be confirmed. In this regard, the French ministries of agriculture, ecology and research and the national research agency started in 2018 the four-year "ROSE Challenge". The Challenge addresses the weed killing of intra row of crops by autonomous robots. Four competing teams tackle the problem by developing autonomous solutions, which will be formally assessed throughout the Challenge. This paper offers an overview of robotics challenges and presents the methodology of the real-life evaluation performed in the context of the ROSE Challenge. The overall object of this contribution aims at raising awareness in the agricultural community about the effectiveness of challenges to support innovation in the agricultural field.