Objectives. We examined the association between individual and clustered lifestyle behaviors in middle age and later in cognitive functioning. Methods. Middle-aged participants (n = 2430) in the Supplementation en Vitamines et Mineraux Antioxydant study self-reported their low physical activity, sedentary behavior, alcohol use, smoking, low fruit and vegetable consumption, and low fish consumption. We assessed cognition 13 years later via 6 neuropsychological tests. After standardization, we summed the scores for a composite cognitive measure. We estimated executive functioning and verbal memory scores using principal component analysis. We estimated the mean differences (95% confidence intervals [CIs]) in cognitive performance by the number of unhealthy behaviors using analysis of covariance. We identified latent unhealthy behavior factor via structural equation modeling. Results. Global cognitive function and verbal memory were linearly, negatively associated with the number of unhealthy behaviors: adjusted mean differences = -0.36 (95% CI = -0.69, -0.03) and -0.46 (95% CI = -0.80, -0.11), respectively, per unit increase in the number of unhealthy behaviors. The latent unhealthy behavior factor with low fruit and vegetable consumption and low physical activity as main contributors was associated with reduced verbal memory (RMSEA = 0.02; CFI = 0.96; P = .004). No association was found with executive functioning. Conclusions. Comprehensive public health strategies promoting healthy lifestyles might help deter cognitive aging.