Counterfactual reasoning aims at predicting how the world would have been had a certain event occurred, and as such has attracted attention from the fields of explainability and robustness in machine learning. While Pearl's causal inference provides appealing rules to calculate valid counterfactuals, it relies on a model that is unknown and hard to discover in practice. We formalize a mass transportation viewpoint of counterfactual reasoning and use distributional matching methods as a natural model-free surrogate approach. In particular, we show that optimal transport theory defines relevant counterfactuals, as they are numerically feasible, statistically-faithful, and can even coincide with counterfactuals generated by linear additive causal models. We argue this has consequences for interpretability and we illustrate the strength of the mass transportation viewpoint by recasting and generalizing the accepted counterfactual fairness condition into clearer, more practicable criteria.