Colorectal cancer (CRC) is the third most common cause of cancer-related death. Significant improvements in CRC treatment have been made for the last 20 years, on one hand thanks to a better detection, allowing surgical resection of the incriminated area, and on the other hand, thanks to a better knowledge of CRC’s development allowing the improvement of drug strategies. Despite this crucial progress, CRC remains a public health issue. The current model for CRC initiation and progression is based on accumulation of sequential known genetic mutations in the colon epithelial cells’ genome leading to a loss of control over proliferation and survival. However, increasing evidence reveals that CRC initiation is more complex. Indeed, chronic inflammatory contexts, such as inflammatory bowel diseases, have been shown to increase the risk for CRC development in mice and humans. In this manuscript, we review whether colon fibroblasts can go from the main regulators of the ISC homeostasis, regulating not only the renewal process but also the epithelial cells’ differentiation occurring along the colon crypt, to the main player in the initiation of the colorectal cancer process due to chronic inflammation.