Estimating genetic trends using historical data is an important parameter to check the success of the breeding programs. The estimated genetic trends can act as a guideline to target the appropriate breeding strategies and optimize the breeding program for improved genetic gains. In this study, 17 years of historical data from IRRI’s rice drought breeding program was used to estimate the genetic trends and assess the breeding program's success. We also identified top-performing lines based on grain yield breeding values as an elite panel for implementing future population improvement-based breeding schemes. A two-stage approach of pedigree-based mixed model analysis was used to analyze the data and extract the breeding values and estimate the genetic trends for grain yield under non-stress, drought, and in combined data of non-stress and drought. Lower grain yield values were observed in all the drought trials. Heritability for grain yield estimates ranged between 0.20 and 0.94 under the drought trials and 0.43–0.83 under non-stress trials. Under non-stress conditions, the genetic gain of 0.21% (10.22 kg/ha/year) for genotypes and 0.17% (7.90 kg/ha/year) for checks was observed. The genetic trend under drought conditions exhibited a positive trend with the genetic gain of 0.13% (2.29 kg/ha/year) for genotypes and 0.55% (9.52 kg/ha/year) for checks. For combined analysis showed a genetic gain of 0.27% (8.32 kg/ha/year) for genotypes and 0.60% (13.69 kg/ha/year) for checks was observed. For elite panel selection, 200 promising lines were selected based on higher breeding values for grain yield and prediction accuracy of > 0.40. The breeding values of the 200 genotypes formulating the core panel ranged between 2366.17 and 4622.59 (kg/ha). A positive genetic rate was observed under all the three conditions; however, the rate of increase was lower than the required rate of 1.5% genetic gain. We propose a recurrent selection breeding strategy within the elite population with the integration of modern tools and technologies to boost the genetic gains in IRRI’s drought breeding program. The elite breeding panel identified in this study forms an easily available and highly enriched genetic resource for future recurrent selection programs to boost the genetic gains.