The structural pattern of infectivity matrices, where a set of parasites is confronted to a set of hosts, is a key parameter for our understanding of biological interactions and their evolution. This pattern determines evolution of parasite pathogenicity and host resistance, the spatio-temporal distribution of host and parasite genotypes and the efficiency of disease control strategies. Two major patterns have been proposed for plant-virus genotypes infectivity matrices. In the gene-for-gene model, infectivity matrices show a nested pattern, where the host range of specialist virus genotypes is a subset of the host range of less-specialized viruses. In contrast, in the matching-allele (MA) model, each virus genotype is specialized to infect one (or a small set of) host genotype(s). The corresponding infectivity matrix shows a modular pattern, where infection is frequent for plants and viruses belonging to the same module but rare for those belonging to different modules. We analyzed the structure of infectivity matrices between Potato virus Y (PVY) and plant genotypes in the family Solanaceae carrying different eukaryotic initiation factor 4E (eIF4E)-coding alleles conferring recessive resistance. Whereas this system corresponds mechanistically to a MA model, the expected modular pattern was rejected for our experimental data. This was mostly because PVY mutations involved in adaptation to a particular plant genotype displayed frequent pleiotropic effects, conferring simultaneously an adaptation to additional plant genotypes carrying different eIF4E alleles. Such effects should be taken into account for the design of strategies of sustainable control of PVY through plant varietal mixtures or rotations.