The plant Silene latifolia has separate sexes and sex chromosomes, and is of interest for studying the early stages of sex chromosome evolution, especially the evolution of non-recombining regions on the Y chromosome. Hitch-hiking processes associated with ongoing genetic degeneration of the non-recombining Y chromosome are predicted to reduce Y-linked genes' effective population sizes, and S. latifolia Y-linked genes indeed have lower diversity than X-linked ones. We tested whether this represents a true diversity reduction on the Y, versus the alternative possibility, elevated diversity at X-linked genes, by collecting new data on nucleotide diversity for autosomal genes, which had previously been little studied. We find clear evidence that Y-linked genes have reduced diversity. However, another alternative explanation for a low Y effective size is a high variance in male reproductive success. Autosomal genes should then also have lower diversity than expected, relative to the X, but this is not found in our loci. Taking into account the higher mutation rate of Y-linked genes, their low sequence diversity indicates a strong effect of within-population hitch-hiking on the Y chromosome.