In Sub-Saharan Africa, smallholder farms play a key role in agriculture, occupying most of the agricultural land. Design policies for increasing smallholder productivity remains a safe way to establish sustainable food systems and boost local economies. However, efforts are still needed in order to achieve accurate and timely monitoring in smallholder farming systems. With the advent of modern Earth Observation programmes such as the Sentinel satellites, which provide quasi-synchronous and high-resolution multi-source information over any area of the continental surfaces, new opportunities are opened up to accurately map crop yields in smallholder farming systems. This study intends to estimate and forecast millet yields in central Senegal, making the use of multi-source (synthetic-aperture radar (SAR) and optical) image time series and state-of-the-art machine learning models. A Random Forest (RF) model explained up to 50% of the millet yield variability, while deep learning models such as Convolutional Neural Network (CNN) showed promise results but performed lower. We also found that the concatenation of SAR polarizations and vegetation indices improved our crop yield modelling, but such improvement was tightly related to the modelling approach, namely RF and CNN. Using RF to forecast millet yields, we achieved stable and satisfactory accuracy 2 weeks before the harvest period.