We evaluated the effects of three rodent diets differing in soybean meal content on the response of the seminal vesicles, prostate and bulbocavernosus/levator ani (BC/LA) muscle to androgens and anti-androgenic compounds in the Hershberger assay. The diets tested were (1) L5, a semi-synthetic phytoestrogen-free diet, (2) DO4, 8.5% (w/w) vegetable protein and (3) DO3, 22.5% (w/w) vegetable protein. We determined the effects of dietary soy isoflavones after ten days of exposure and in animals fed L5 and DO3 diets throughout their lifetime (including the period of treatment with androgenic or anti-androgenic compounds). After ten days of exposure, we observed no effect of diet on the accessory sex organs of male Wistar rats. In contrast, diet affected the androgenic response to testosterone propionate in seminal vesicles and prostate. Seminal vesicles were the most sensitive organs. Vinclozolin caused a dose-dependent decrease in the relative weights of seminal vesicles, prostate and BC/LA regardless of diet. As vegetable proteins may contain high proportions of genistein and daidzein, two well-known oestrogenic endocrine disrupters that may alter the results of reproductive studies, we recommend the use of a standardised open-formula diet without soy isoflavones, such as L5, if the Hershberger assay is to be performed.