Small populations establishing on colonization fronts have to adapt to novel environments with limited genetic variation. The pace at which they can adapt, and the influence of genetic variation on their success, are key questions for understanding intraspecific diversity. To investigate these topics, we performed a reciprocal transplant experiment between two recently founded populations of brown trout in the sub-Antarctic Kerguelen Islands. Using individual tagging and genetic assignment methods, we tracked the fitness of local and foreign individuals, as well as the fitness of their offspring over two generations. In both populations, although not to the same extent, gene flow occurred between local and foreign gene pools. In both cases, however, we failed to detect obvious footprints of local adaptation (which should limit gene flow) and only weak support for genetic rescue (which should enhance gene flow). In the population where gene flow from foreign individuals was low, no clear differences were observed between the fitness of local, foreign, and F1 hybrid individuals. In the population where gene flow was high, foreign individuals were successful due to high mating success rather than high survival, and F1 hybrids had the same fitness as pure local offspring. These results suggest the importance of considering sexual selection, rather than just local adaptation and genetic rescue, when evaluating the determinants of success in small and recently founded populations.