We introduce an epidemic model with varying infectivity and general exposed and infectious periods, where the infectivity of each individual is a random function of the elapsed time since infection, those function being i.i.d. for the various individuals in the population. This approach models infection-age dependent infectivity, and extends the classical SIR and SEIR models. We focus on the infectivity process (total force of infection at each time), and prove a functional law of large number (FLLN). In the deterministic limit of this LLN, the infectivity process and the susceptible process are determined by a two-dimensional deterministic integral equation. From its solutions, we then derive the exposed, infectious and recovered processes, again using integral equations. For the early phase, we study the stochastic model directly by using an approximate (non--Markovian) branching process, and show that the epidemic grows at an exponential rate on the event of non-extinction, which matches the rate of growth derived from the deterministic linearized equations. We also use these equations to derive the basic reproduction number $R_0$ during the early stage of an epidemic, in terms of the average individual infectivity function and the exponential rate of growth of the epidemic.