Pea-based ingredients are increasingly being used in foods because of their nutritional, functional and environmental benefits. However, their bitter taste is not appreciated by consumers. Saponins have been reported to be bitter in whole pea flour (PF) but not in the purified ingredients obtained from it, such as pea protein isolate (PPI) and pea starch (PS). In addition, the evolution of saponins in cooked foods made from these ingredients and their relationship to bitter flavor has not been investigated. This study, therefore, explored the presence of two bitter saponins, βg and Bb, in whole pea flour (PF) and a composite flour reconstructed from the two main fractions (PS + PPI). In addition, it investigated the impact of baking on the chemical state of these compounds in a sponge cake. Finally, the sensory impact of the baking process on the perceived bitterness of cakes made with these two pea flours was also evaluated. High-Performance Liquid Chromatography–High-Resolution Mass Spectrometry (HPLC-HRMS) was used to identify and quantify pea saponins in the flours and cakes, and a descriptive sensory analysis was obtained by a trained panel to assess sensory differences in bitterness. Our results showed marked differences in saponin concentration and composition among the pea ingredients studied. Concentrations were highest in PPI (1.497 mg·g−1 dry matter), with 98% of saponin Bb. PS had the lowest saponin concentration (0.039 mg·g−1 dry matter, with 83% Bb), while 0.988 mg·g−1 dry matter was quantified in PF, with only 20% Bb and 80% βg. This research also highlighted the thermal degradation of saponin βg to Bb in sponge cakes during baking at 170 °C. However, at a sensory level, these chemical changes were insufficient for the impact on bitterness to be perceived in cakes made with pea flour. Moreover, baking time significantly reduced the bitter flavor in cakes made with the composite flour (PS + PPI).