Mastitis is one of the greatest issues for the global dairy industry and controlling these infections by vaccination is a long-sought ambition that has remained unfulfilled so far. In fact, gaps in knowledge of cell-mediated immunity in the mammary gland (MG) have hampered progress in the rational design of immunization strategies targeting this organ, as current mastitis vaccines are unable to elicit a strong protective immunity. The objectives of this article are, from a comprehensive and critical review of available literature, to identify what characterizes adaptive immunity in the MG of ruminants, and to derive from this analysis research directions for the design of an optimal vaccination strategy. A peculiarity of the MG of ruminants is that it does not belong to the common mucosal immune system that links the gut immune system to the MG of rodents, swine or humans. Indeed, the MG of ruminants is not seeded by lymphocytes educated in mucosal epithelia of the digestive or respiratory tracts, because the mammary tissue does not express the vascular addressins and chemokines that would allow the homing of memory T cells. However, it is possible to elicit an adaptive immune response in the MG of ruminants by local immunization because the mammary tissue is provided with antigen-presenting cells and is linked to systemic mechanisms. The optimal immune response is obtained by luminal exposure to antigens in a non-lactating MG. The mammary gland can be sensitized to antigens so that a local recall elicits neutrophilic inflammation and enhanced defenses locally, resulting from the activation of resident memory lymphocytes producing IFN-γ and/or IL-17 in the mammary tissue. The rational exploitation of this immunity by vaccination will need a better understanding of MG cell-mediated immunity. The phenotypic and functional characterization of mammary antigen-presenting cells and memory T cells are amongst research priorities. Based on current knowledge, rekindling research on the immune cells that populate the healthy, infected, or immunized MG appears to be a most promising approach to designing efficacious mastitis vaccines.