The present study aimed to use comparative genomics to explore the relationships between Frankia and actinorhizal plants using a data set made of 33 Frankia genomes. The determinants of host specificity were first explored for “Alnus-infective strains” (i.e., Frankia strains belonging to Cluster Ia). Several genes were specifically found in these strains, including an agmatine deiminase which could possibly be involved in various functions as access to nitrogen sources, nodule organogenesis or plant defense. Within “Alnus-infective strains”, Sp+ Frankia genomes were compared to Sp− genomes in order to elucidate the narrower host specificity of Sp+ strains (i.e., Sp+ strains being capable of in planta sporulation, unlike Sp− strains). A total of 88 protein families were lost in the Sp+ genomes. The lost genes were related to saprophytic life (transcriptional factors, transmembrane and secreted proteins), reinforcing the proposed status of Sp+ as obligatory symbiont. The Sp+ genomes were also characterized by a loss of genetic and functional paralogs, highlighting a reduction in functional redundancy (e.g., hup genes) or a possible loss of function related to a saprophytic lifestyle (e.g., genes involved in gas vesicle formation or recycling of nutrients).