Remote sensing is a most promising technique for providing crop maps, thanks to the development of satellite images at various temporal and spatial resolutions. Three-dimensional (3D) convolutional neural networks (CNNs) have the potential to provide rich features that represent the spatial and temporal patterns of crops when applied to time series. This study presents a novel 3D-CNN framework for classifying crops that is based on the fusion of radar and optical time series and also fully exploits 3D spatial-temporal information. To extract deep convolutional maps, the proposed technique uses one separate sequence for each time series dataset. To determine the label of each pixel, the extracted feature maps are passed to the concatenating layer and subsequent transmitted to the sequential fully connected layers. The proposed approach not only takes advantage of CNNs, i.e. automatic feature extraction, but also discovers discriminative feature maps in both, spatial and temporal dimensions and preserves the growth dynamics of crop cycles. An overall accuracy of 91.3% and a kappa coefficient of 89.9% confirm the proposed method's potential. It is also shown that the suggested approach outperforms other methods.