The choice of tree species planted with cacao trees is essential for ensuring the efficiency and sustainability of cacao farming systems. This raises the question of the long-term impact of associated tree legumes (ATLs) on cacao bean yields and biomass accumulation. This study was carried out in accordance with four-block randomised experimental design in Divo (Ivory Coast, West Africa). The study involved cacao-Albizia lebbeck (Cacao-Alb) and cacao-Acacia mangium (Cacao-Aca) intercrops and unshaded cacao plots (Control). After 20 years of intercropping, we assessed cacao dendrometry (height, circumference, biomass, and biomass C-stock) and production (number of pods per tree and bean yield at the plot level) as well as soil fertility (soil organic matter (SOM) concentration) at various distances from ATLs (D1:0–1.75 m; D2:3.25–5 m; D3:7–9 m). The distance from the ATLs had no significant effect on the measured cacao dendrometric parameters, except for cacao height. In contrast, the Cacao-Aca association had a negative impact on the SOM concentration (−22%), cacao tree height (−6.15%), and productivity parameters (biomass: −12.4%; bean yield: −43%). However, Cacao-Alb and the Control had no significant differences in terms of SOM, tree biomass, or bean production. Intercropping cacao with the tested tree legumes did not enhance cacao productivity and, in some cases, hindered it, depending on the ATL species. This study highlighted the importance of identifying appropriate shade tree legume species that could be promoted in cacao-based agroforestry systems.