During the development of tissues, complex programs take place to reach terminally differentiated states with specific gene expression profiles. Epigenetic regulations such as histone modifications and chromatin condensation have been implicated in the short and long-term control of transcription. It has recently been shown that the 3D spatial organization of chromosomes in the nucleus also plays a role in genome function. Indeed, the eukaryotic interphase nucleus contains sub-domains that are characterized by their enrichment in specific factors such as RNA Polymerase II, splicing machineries or heterochromatin proteins which render portions of the genome differentially permissive to gene expression. The positioning of individual genes relative to these sub-domains is thought to participate in the control of gene expression as an epigenetic mechanism acting in the nuclear space. Here, we review what is known about the sub-nuclear organization of mammary epithelial cells in connection with gene expression and epigenetics. Throughout differentiation, global changes in nuclear architecture occur, notably with respect to heterochromatin distribution. The positions of mammary-specific genes relative to nuclear sub-compartments varies in response to hormonal stimulation. The contribution of tissue architecture to cell differentiation in the mammary gland is also seen at the level of nuclear organization, which is sensitive to microenvironmental stimuli such as extracellular matrix signaling. In addition, alterations in nuclear organization are concomitant with immortalization and carcinogenesis. Thus, the fate of cells appears to be controlled by complex pathways connecting external signal integration, gene expression, epigenetic modifications and chromatin organization in the nucleus.