Most of the resonant photonic structures behave like open cavities for light where light is trapped for some time before leaking or being absorbed. Their modes are called quasi-normal modes and are associated with complex eigenfrequencies omega(n) = omega(n)' + i omega(n)'', omega(n)'' characterizing the rate at which the energy leaks from the structure. As a consequence, one can show that the fields of these modes diverge far away from the scatterer. This is problematic when one attempts to develop a theoretical description of the resonant interaction between light and resonant photonic stuctures in terms of their quasi-normal modes. Moreover, the existence or not of a non-resonant term in addition to these resonant contributions is still an open problem. Here, we address these two problems by deriving pole-expansions of the scattering operators of resonant optical structures. We evince the existence of a non-resonant term and we solve the problem of the divergence by studying the scattered field in the time domain and by using the causality principle. The quasi-normal mode expansion that we obtain will be of a great use to study light-matter interactions since it allows to determine the optical response of a photonic resonator both in the time and frequency domain.