We study the impact of a hard selective sweep on the genealogy of partially linked neu-tral loci in the vicinity of the positively selected allele. We consider a sexual population of varying size and, focusing on two neighboring loci, derive an approximate formula for the neutral geneal-ogy of a sample of individuals taken at the end of the sweep. We prove that the stochasticity of the population size has a major impact on this genealogy and apply our results to the derivation of the linkage disequilibrium between neutral loci, a statistic which is frequently used to detect recent se-lection on genetic data. Furthermore, we provide a deeper insight into the dynamics of the mutant and wild type population during the different stages of a selective sweep.