There is increasing interest in sex-specific nongenetic and noncultural mechanisms transferring a memory of parental exposure to various environments and determining the reactivity of subsequent generations to their environment during their lifetimes. The epigenetic origins of sexual dimorphism are due not only to the chromosomal sex (XX or XY) at conception prior to gonad differentiation, but later on, to a complex intermingling of both hormones and X and Y genes regulating autosomal genes. Sexual dimorphism may also date back to the sex of ancestors who experienced exposure to different environmental factors and transmitted their “imprint” in a sex-specific manner, either through the maternal or paternal lineage or both, and in a manner that may also depend on the sex of the offspring. Observed sexual dimorphism is the result of a subtle entanglement of different epigenetic and other factors and how and when early and life-long environmental exposures interact with them, leading to specific responses, outcomes, and persistence for men and women.