Vegetation cover estimation for overstory and understory layers provides valuable information for modeling forest carbon and water cycles and refining forest ecosystem function assessment. Although previous studies demonstrated the capability of light detection and ranging (LiDAR) in the three-dimensional (3D) characterization of forest overstory and understory communities, the high cost inhibits its application in frequent and successive survey tasks. Low-cost commercial red-green-blue (RGB) cameras mounted on unmanned aerial vehicles (UAVs), as LiDAR alternatives, provide operational systems for simultaneously quantifying overstory crown cover (OCC) and understory vegetation cover (UVC). We developed an effective method named back-projection of 3D point cloud onto superpixel-segmented image (BAPS) to extract overstory and forest floor pixels using 3D structure-from-motion (SfM) point clouds and two-dimensional (2D) superpixel segmentation. The OCC was estimated from the extracted overstory crown pixels. A reported method, called half-Gaussian fitting (HAGFVC), was used to segement green vegetation and non-vegetation pixels from the extracted forest floor pixels and derive UVC. The UAV-based RGB imagery and field validation data were collected from eight forest plots in Saihanba National Forest Park (SNFP) plantation in northern China. The consistency of the OCC estimates between BAPS and canopy height model (CHM)-based methods (coefficient of determination: 0.7171) demonstrated the capability of the BAPS method in the estimation of OCC. The segmentation of understory vegetation was verified by the supervised classification (SC) method. The validation results showed that the OCC and UVC estimates were in good agreement with reference values, where the root-mean-square error (RMSE) of OCC (unitless) and UVC (unitless) reached 0.0704 and 0.1144, respectively. The low-cost UAV-based observation system and the newly developed method are expected to improve the understanding of ecosystem functioning and facilitate ecological process modeling.