Send to

Choose Destination
Hum Hered. 2016;82(1-2):37-49. doi: 10.1159/000477782. Epub 2017 Aug 3.

Estimation of FST and the Impact of de novo Mutation.

Author information

Center for Research on Genomics and Global Health, National Human Genome Research Institute, Bethesda, MD, USA.



Wright defined FST as a measure of genetic differentiation. Cockerham developed an estimator of FST based on binary indicators in an ANOVA framework. Here, we address 2 issues regarding the estimation of FST. First, we derive a new estimator of FST based on the ANOVA framework using the doubly truncated normal distribution as an approximation of the binomial distribution to estimate variances. Second, we consider the impact of de novo mutation on FST estimation.


We compare our estimator to Weir and Cockerham's estimator via computer simulation. We apply our estimator to whole genome sequence data from the 1000 Genomes Project. We use chimpanzee whole genome sequence data to ascertain for ancestral polymorphisms.


By simulation, our new estimator is less biased than Weir and Cockerham's estimator for comparison of two subpopulations and is systematically more precise. As determined empirically by ascertainment of ancestral polymorphisms and theoretically, the effect of de novo mutation on FST estimation with human whole genome sequence data is statistically negligible. The effect of down-sampling ancestral polymorphisms is also statistically negligible.


These results improve and simplify the use and interpretation of FST in studies of population structure.


Bias; Genetic differentiation; Population structure; Precision


Supplemental Content

Full text links

Icon for S. Karger AG, Basel, Switzerland
Loading ...
Support Center