Most seed plants produce both pollen and ovules. In principle, pollen export could interfere with pollen import through self-pollination, resulting in ovule usurpation and reduced fruit set. Evidence for such interference exists under experimental settings but its importance under natural conditions is unknown. To test for sexual interference in nature, it is necessary to study together mating system, through paternity analyses, and fruit set, the proportion of flowers giving seeds or fruits. We developed a new model combining both processes, using chestnut (Castanea) as case study. We carried out a paternity analysis in an intensively studied plot of 273 trees belonging to three interfertile chestnut species and including a range of individuals with more or less functional stamens, resulting in a large data set of 1924 mating events. We then measured fruit set on 216 of these trees. Fruit set of male-fertile trees was much lower than that of male-sterile trees. Our process-based model shows that pollen is not limiting in the study site and hence cannot account for reduced fruit set. It also indicates that self-pollination is high (74%) but selfing rate is low (4%). Self-pollen is less competitive than cross-pollen, reducing sexual interference, but not sufficiently, as many ovules end up being self-fertilized, 95% of which abort before fruit formation, resulting in the loss of 46% of the fruit crop. These results suggest that the main cause of reduced reproductive potential in chestnut is sexual interference by self-pollen, raising questions on its evolutionary origins.