Contemporary genetic data has been extensively used to infer demographic and adaptive history of populations. However, those inferences integrate effects over large periods of time and are often uninformative about the recent events that will be more relevant for the management of populations. In contrast, temporal genetic data, such as those obtained from ancient and modern samples or from monitoring surveys, can provide information on the recent evolutionary history of the target population. At present, there are some statistical methods available to make inferences from temporal population genetic data, but most of them suffer from two limitations: (1) demographic inference ignores the effects of linked selection which, in some cases, can produce biases in the inference; and (2) inference of selection focuses on loci with large effects that produce outlier patterns but that provide little information about the adaptive potential and viability of the population. In this work we propose a simulation-based approach (approximate Bayesian computation, ABC) to address those limitations. Using individual-based forward in time simulations we are able to model multi-locus selection processes and their effect on whole-genome diversity. In each simulation, latent variables (effective population size and genetic load) are calculated that integrate demographic and selective information in a way that is not captured by the original parameters of the model (e.g. census population size). Simulations are used to generate a training data set and Random Forest ABC are used to learn about the demographic and selective parameters and variables from the genetic diversity patterns. The performance of the inference is evaluated via out-of-bag estimates. The results show this is a promising approach for the joint inference of demography and selection. Inference of effective population size is accurate even in scenarios with pervasive selection were naive estimators show significant bias. As an example, the method is applied to a data set of feral populations of European bee in North America (modern samples and museum specimens), with results congruent with the known biology of the species.