Colletotrichum gloeosporioides is a species complex of agricultural importance as it causes anthracnose disease on many crop species worldwide, and strong impact regionally on Water Yam (Dioscorea alata) in the Caribbean. In this study, we conducted a genetic analysis of the fungi complex in three islands of the Lesser Antilles—Guadeloupe (Basse Terre, Grande Terre and Marie Galante), Martinique and Barbados. We specifically sampled yam fields and assessed the genetic diversity of strains with four microsatellite markers. We found a very high genetic diversity of all strains on each island, and intermediate to strong levels of genetic structure between islands. Migration rates were quite diverse either within (local dispersal) or between islands (long-distance dispersal), suggesting important roles of vegetation and climate as local barriers, and winds as an important factor in long-distance migration. Three distinct genetic clusters highlighted different species entities, though there was also evidence of frequent intermediates between two clusters, suggesting recurrent recombination between putative species. Together, these results demonstrated asymmetries in gene flow both between islands and clusters, and suggested the need for new approaches to anthracnose disease risk control at a regional level.