Carotenoid pigments signature in the fat using visible reflectance spectroscopy has shown high potential for distinguishing pasture-fed (P) from stall concentrate-fed (S) lamb carcasses. However, a recent study demonstrated a between-breed variability in the digestive and metabolic fate of carotenoids pigments. The present study was therefore designed to investigate the extent to which this between-breed variability may affect the reliability of diet authentication using visible spectroscopy of the fat. We used 1054 male lambs from three breeds (Romane (ROM), Ile-de-France (OIF) and Limousine (LIM)). The breed-feed breakdown was 148 P and 258 S ROM, 102 P and 92 S OIF and 168 P and 286 S LIM lambs. The reflectance spectrum of perirenal fat was measured at 24 h postmortem at wavelengths between 400 and 700 nm. We quantified light absorption in the 450 to 510 nm area by calculating a traceability index (AVMI(450 to 510)) considered as an indicator of the carotenoid concentration in the fat (method 1) and we performed a multivariate analysis over the full set of reflectance data between 400 and 700 nm (method 2). The reliability of method 1 proved very variable across breeds, with a percentage of correctly classified lambs reaching 95.3%, 90.5% and 79.4% in ROM, LIM and OIF lambs, respectively. Despite these between-breeds differences, the threshold of the linear discriminant analysis performed on AVMI(450 to 510) was fairly similar between breeds; when all the data for the three breeds were pooled, the threshold cut-off value was 224 units and the method correctly classified 90.2% of the 1054 lambs. Using the full range of reflectance data (method 2) enabled to significantly increase the proportion of correctly classified lambs for both OIF and LIM breeds, but not for ROM breed. It enabled to correctly classify 96.1%, 94.5% and 94.8% of the ROM, LIM and OIF lambs. The reliability of the discrimination was not significantly different when pooling all lambs for the three breeds than when using a breed-specific database (93.9% and 95.2%, respectively).