Understanding the long-term impacts of invasive mammalian browsers and granivores in mixed forests is difficult due to the many processes potentially affecting the demography of long-lived trees. We constructed individual-based spatially-explicit simulation models of two mixed conifer–angiosperm forests, growing on soils of contrasting phosphorus (P) availability, to investigate how browsing by invasive red deer (Cervus elaphus scoticus) and granivory by invasive rodents (primarily house mouse Mus musculus) might alter forest dynamics. Models were parameterized with field data. Seedling growth and survival rates were estimated inside and outside deer exclosures. Seed predation rates were estimated at high and low rodent densities in mast and non-mast seeding years. For the alluvial terrace forest, which grew on P-rich soil, our model contained 15 tree species dominated by angiosperms; our model of the P-poor marine terrace forest contained 7 species dominated by conifers. The two forest models were used to explore four 500-year scenarios: deer and rodents present, deer present and rodents absent, deer absent and rodents present, and deer and rodents absent. Our field studies revealed that the highest probabilities of seed predation by rodents occurred for two canopy species in mast years and in neighborhoods of high basal area, and that deer browsing had the greatest negative effects on the growth and survival rates of angiosperm seedlings. Our simulation models predicted that the presence of deer and rodents would reduce the abundances of canopy co-dominants, and hence total basal area, in the alluvial terrace forest. The presence of deer increased the dominance of conifers in both forests, but the effects of deer and rodents were much stronger in the alluvial terrace forest. Our study revealed two emergent properties. First, rodent- and deer-induced reductions in seedling abundances did not always translate into reduced sapling and adult tree abundances. Second, when deer changed the abundance of a canopy dominant, other species were affected by altered interspecific competition. We conclude that the effects of invasive deer and rodents will be greatest in forests growing on P-rich soils. Exclusion of deer is predicted to have greater long-term effects on forest dynamics than exclusion of rodents.