Background Physical activity (PA) and sedentary behavior have inconsistent effects on fertility. High body mass index is associated with infertility but to our knowledge, very few studies have explored body composition in association to fertility. Objective To assess the association between physical inactivity, sedentary behavior, body composition and idiopathic infertility in French men and women. Design We conducted a case-control multicentric observational study. 159 infertile (79 men and 80 women) and 143 fertile (72 men and 71 women) were recorded in four fertility centers. Main outcome measures Participants completed self-administered questionnaires on sociodemographic and lifestyle characteristics, dietary intake, physical activity and sedentary behavior. Anthropometrics were measured, and bioelectrical impedance analysis was used to estimate body composition. Multivariable logistic regression was used to analyze the association of fertility with PA level and sedentary behavior. Results In men, being physically inactive (Odd ratio [OR] 2.20; 95% confidence interval [CI], 1.06, 4.58) and having fat mass greater than the reference values for their age (OR 2.83; 95% CI, 1.31, 6.10) were positively associated with infertility. Sedentary behavior and fat-free mass were not related to infertility in men. In women, sedentary behavior (OR 3.61; 95% CI, 1.58, 8.24), high body fat (OR 3.16; 95% CI, 1.36, 7.37) and low fat-free mass (OR 2.65; 95% CI, 1.10, 6.37) were associated with infertility. PA level was not associated with fertility in women. Conclusions This study suggests that sedentary behavior and physical inactivity would represent two independent risk factors associated with infertility. The various elements that make up physical activity (frequency, intensity, time, and type of exercise) and the interrupting time spent sitting should be considered. Body composition variation should be explored further in relation to the biological pathways involved in idiopathic infertility. Moreover, the improvement of lifestyle factors should be considered in infertility treatment.