Curr Biol. 2007 Dec 4; 17(23): 2061–2066.
PMCID: PMC2148465

The Genetic Basis of Inbreeding Avoidance in House Mice


Animals might be able to use highly polymorphic genetic markers to recognize very close relatives and avoid inbreeding [1, 2]. The major histocompatibility complex (MHC) is thought to provide such a marker [1, 3–6] because it influences individual scent in a broad range of vertebrates [6–10]. However, direct evidence is very limited [1, 6, 10, 11]. In house mice (Mus musculus domesticus), the major urinary protein (MUP) gene cluster provides another highly polymorphic scent signal of genetic identity [8, 12–15] that could underlie kin recognition. We demonstrate that wild mice breeding freely in seminatural enclosures show no avoidance of mates with the same MHC genotype when genome-wide similarity is controlled. Instead, inbreeding avoidance is fully explained by a strong deficit in successful matings between mice sharing both MUP haplotypes. Single haplotype sharing is not a good guide to the identification of full sibs, and there was no evidence of behavioral imprinting on maternal MHC or MUP haplotypes. This study, the first to examine wild animals with normal variation in MHC, MUP, and genetic background, demonstrates that mice use self-referent matching of a species-specific [16, 17] polymorphic signal to avoid inbreeding. Recognition of close kin as unsuitable mates might be more variable across species than a generic vertebrate-wide ability to avoid inbreeding based on MHC.

Keywords: EVO_ECOL


Our experimental design met a set of stringent requirements to establish whether mice use major histocompatibility complex (MHC) and/or major urinary protein (MUP) to avoid inbreeding. First, genetic recognition and mate preferences need to be demonstrated against normally variable genetic and environmental backgrounds typical of natural populations. We therefore used wild house mice (Mus musculus domesticus) rather than genetically homogeneous laboratory mice that are hybrids of three Mus subspecies derived from an extremely small pool of founders [18, 19] and have been subject to strong artificial selection for ease of breeding in the laboratory [4, 20]. Second, previous studies examining kin recognition and inbreeding avoidance in mice have used only MHC types derived from laboratory strains (e.g., [3–5, 21, 22]). The role of MUPs in this context has not been examined previously, probably because of the lack of variation in MUP patterns within and between laboratory strains ([23], J.L.H. and R.J.B., unpublished data, and S.A. Cheetham, personal communication). This is in dramatic contrast to wild house mice, in which individual variation in the pattern of urinary MUPs is the main basis for individual recognition through scent [12, 13, 15, 24]. To properly reflect natural variation at these highly polymorphic gene complexes, we used founder mice that carried a large range of MHC and MUP haplotypes derived directly from wild animals. Third, in natural populations, animals that share the same genotype at a highly polymorphic marker such as MHC or MUP will be closely related and share many other alleles across the genome. In order to test whether inbreeding avoidance is driven by the sharing of MHC and/or MUP haplotypes (inherited independently) on naturally variable genetic backgrounds, we controlled for background relatedness in our analysis and design and thus the sharing of other alleles across the genome. Finally, to ensure natural behavior and mate choice, we allowed mice to breed freely in very large seminatural enclosures without intervention or disturbance.

In house mouse populations, dominant males defend small territories in which they monopolize mating opportunities [25], and extraterritorial matings by females with neighbor territory owners are frequent [4, 26]. Animals that remain in the same local population are thus likely to encounter unfamiliar half sibs that share the same father but different mothers in addition to full sibs that share both father and mother. To model this, we released four replicate populations of half-sib and full-sib outbred house mice (each derived from one father and several unrelated mothers with full sibs reared only by their mothers; see the Supplemental Experimental Procedures in the Supplemental Data available online) into four very large seminatural enclosures (each 250 m2 with abundant cover and food). Tissue samples for genotyping were obtained from all F1 founders (33 female, 48 male) prior to release and from their parents, allowing us to establish the separate MHC and MUP haplotypes of each heterozygous founder. The combinations of males and females available as potential mating partners covered the full range of possible MHC and MUP haplotype sharing (Table S1). After 15 weeks, when founding females could have reared up to three litters to independence, animals were captured from each population and F2 offspring of the original founders (n = 483) were genotyped with 40 microsatellite markers so that parentage and MHC and MUP haplotypes could be independently established. From these data, we determined for each F1 founder female the minimum number of successful matings with each available male (based on recovered independent offspring) and the number of offspring captured from each female-male pair.

Consistent with inbreeding avoidance, we found a deficit in the overall frequency of successful matings between full sibs (Table S2), although the effect was too weak to differ significantly from random mating per female (Table 1: multinomial logistic model 1, p = 0.08). We found no evidence for disassortative mating on the basis of MHC haplotype sharing (Table 1: model 2), i.e., there was no evidence for fewer successful matings between mice that shared either one or both MHC haplotypes relative to those that shared no haplotypes (Table S2; Figure S1). By contrast, MUP sharing had a strong and highly significant effect on the likelihood of successful mating (Table 1: model 3, p = 0.005; Figure S1). Specifically, there was no deficit when only one MUP haplotype was shared, but there were many fewer matings between mice that shared both MUP haplotypes (complete match) than expected under random mating conditions (Table 1: model 4, p < 0.002). The same trend was evident across all four populations (Table S1). We confirmed that this effect was due to MUP sharing between males and females rather than inherent differences in male quality because there was no difference in the overall mating success of males represented in full MUP sharing dyads (mean ± standard error of the mean [SEM] matings per male with any female: 4.4 ± 0.6, n = 23) compared to the success of other males (3.8 ± 0.5 matings per male, n = 25; t46 = −0.70, p = 0.49). The strong deficit in successful matings between mice of the same MUP genotype also accounts for the weak deficit in matings between full sibs (Table 1: model 5 versus model 4, p = 0.6) because full sibs were much more likely than half sibs to share both MUP haplotypes (Table S1).

Table 1
Multinomial Logistic Models for Frequency of Mating

It has been hypothesized that familial (behavioral) imprinting on maternal haplotypes would allow animals to avoid inbreeding with a greater proportion of kin than the use of self-haplotype sharing alone [21]. Behavioral imprinting on maternal haplotypes (MHC, MUP, or other genetic loci) should result in a general avoidance of full sibs compared to paternal half sibs because all full sibs carry a maternal haplotype at each potential genetic marker. However, the inclusion of overall relatedness (full versus paternal half sib) as well as MUP genotype sharing in the model provides no better explanation of the likelihood of mating than does MUP genotype alone (Table 1: model 5 versus model 4, p = 0.6). Specific analysis of female behavioral imprinting on maternal MHC haplotypes (Table 1: model 6) or maternal MUP haplotypes (Table 1: model 7) also failed to significantly improve explanation of the likelihood of mating above the direct sharing of MUP genotype alone (see the Supplemental Experimental Procedures for further discussion).

There were fewer offspring per successful mating with full sibs than random expectation (Table 2: model 1, p = 0.02), consistent with the lower viability of offspring due to inbreeding depression [2, 27] or with postcopulatory sperm or embryo selection by females [28]. However, there was no evidence that MHC or MUP sharing between parents per se reduced the viability of offspring from successful matings. Thus, it is unlikely that the strong deficit in successful matings between mice of the same MUP genotype (above) was due to lower viability of these offspring. The small deficit in offspring from matings between animals sharing both MHC haplotypes or both MUP haplotypes (Table S2) was nonsignificant (Table 2) and was due to the fact that most dyads in these two categories were full sibs. Full sibs had significantly fewer offspring per mating regardless of MHC or MUP sharing. Similarly, the MHC and MUP genotypes of all F2 offspring followed the expected pattern of Mendelian inheritance (Table S2), indicating that MHC and MUP homozygotes and heterozygotes had equal viability with no evidence for differential postcopulatory selection.

Table 2
Multinomial Logistic Models for Number of Offspring per Mating

Why should mice avoid mates that share both MUP haplotypes but not those that share one haplotype? In our experiment, the sharing of both MUP haplotypes was a good predictor of whether a potential mate was a full sib or not: 31% of full-sib dyads shared both MUP haplotypes versus only 5% of half-sib dyads (Table S1). However, because 45% of full sibs and 54% of half sibs shared one MUP haplotype, this was a poor guide to whether a potential mate was a full sib. A simulation of populations containing different numbers of haplotypes shows this to be a general rule and not just a consequence of the mix of founders used in the current experiment (Figure S2). Even when there are many haplotypes in the population, the sharing of one or zero haplotypes provides very little information on whether a potential mate is likely to be a full sib or a nonrelative, whereas the sharing of both haplotypes considerably increases the likelihood that animals are full sibs. Mice thus avoid mating when shared MUP type reliably indicates very close relatedness.


The use of MUP alone was sufficient to explain inbreeding avoidance in this study. Although the deficit in mating between mice of the same MUP type was very strong, there was no evidence that animals used genetic markers other than MUP to improve their level of inbreeding avoidance. Further, there was no evidence to support the previously untested hypothesis that animals could increase the range of relatives avoided by imprinting on the separate haplotypes carried by their mother [21]. Such a strategy would also result in the rejection of any unrelated animals that happen to share one maternal haplotype. Notably, animals should only avoid related partners when there is a high inbreeding load [29, 30], which is only likely to occur between very close relatives within outbred populations. As we have shown, the sharing of both haplotypes (but not one) is a reliable indicator of very close relatedness. Nonetheless, in fully outbred populations, most full sibs will not share both MUP haplotypes (only 31% shared both MUP haplotypes in the current study), a situation that applies to any highly polymorphic loci, including MHC. Although this does not exclude a large proportion of close relatives, avoidance between mice sharing both MUP haplotypes might be sufficient to drive widespread avoidance between very close relatives under natural conditions. Theoretical modeling predicts that female discrimination against related males in polygynous mating systems will drive male-biased dispersal from natal areas even when discrimination is relatively weak [29]. Consistent with this, the high level of dispersal among young males in house mouse populations [31] will have a general effect of separating close relatives of the opposite sex, reducing inbreeding even between pairs that do not share MUP type.

The use of MUP as a genetic marker for inbreeding avoidance (whether a pre- or postcopulatory mechanism) will promote genome-wide heterozygosity that includes MHC. Despite widely held assumptions in the literature that MHC-based scents are used by females to avoid inbreeding and promote MHC heterozygosity [1, 5–9, 22, 32], direct evidence is surprisingly limited. Correlations between MHC dissimilarity and mate choice within natural populations [11] might arise through the use of other non-MHC cues because of the normal correlation between MHC and genome-wide similarity. Some congenic laboratory mice that differ only at MHC (though see [7]) show disassortative mating preferences, but others do not [3, 5]. Moreover, the relevance of such studies to the behavior of normal mice is highly questionable [4, 20]. Genetic recognition also needs to be demonstrated against the variable genetic and environmental backgrounds typical of natural populations [15, 33]. Studies of hybrid laboratory mice crossed with wild mice to derive subjects with laboratory-derived MHC haplotypes but with 50% of their genomes from wild mice [4, 21, 34] reveal a deficit of MHC homozygous offspring. However, the design and interpretation of these studies remains controversial [1, 5, 35, 36]. Offspring were typed only for MHC, so parentage could not be assigned and, crucially, parental differences in overall relatedness and other genes that might contribute to inbreeding avoidance could not be assessed. Because the inbred strains used to derive MHC and half of the genome also had two different MUP types, the deficit in MHC homozygous offspring could have arisen from a correlation between MHC and MUP types in the derivation of the founder lines. Genetic variation in these hybrids was also likely to be substantially reduced compared to wild mice. Further research is needed for us to understand whether familial imprinting on MHC, MUP, or other genes is important in much more inbred populations when potential mates might share a complete match to maternal genotypes or perhaps in extreme situations when only MUP identical mates are available. However, such situations could be very unusual because MUP variation persists even in an isolated population with low overall genetic variation that regularly undergoes genetic bottlenecks [37]. Our findings further highlight the limitations of using laboratory strains in mate-selection studies when mice are unable to use their normally variable MUP signals for either kin or individual recognition. Not only will these key signals be unavailable in tests of preference, but the lack of exposure to natural variation in individual scents during the rearing of inbred animals might strongly impact the development of normal recognition processes.

Two broad outcomes derive from this study. First, it challenges the widely held assumption that MHC scents provide a general mechanism across vertebrates to avoid inbreeding and promote MHC heterozygosity. This idea has arisen largely from studies of laboratory mice under extremely abnormal genetic, social, and environmental conditions. Instead, normal wild mice use a set of species-specific urinary proteins to avoid inbreeding that has evolved to provide optimized characteristics for effective signaling through their urine scent-marking system. Given the importance of reliable identity information for mate selection and reproductive success, we should expect animals to evolve signals that are most appropriate for reliable communication in that species. The only known function of MUPs is in scent communication, and they are produced at very high abundance in urine by mice of both sexes but with particularly strong investment by adult males [14]. These proteins provide direct information about genetic identity at several different levels in mice (species, sex, individual, and kinship) ([12, 15, 16, 38], present study) in addition to integrating identity and status information through bound low-molecular-weight volatile pheromones [13, 14, 39]. Individual MUP signatures can easily be identified regardless of other genetic variation between wild mice [12, 15], confirmed by the disruption of recognition when a recombinant MUP is added to wild mouse urine [12]. The fact that mice failed to use MHC scents in addition to MUP when this could have improved inbreeding avoidance suggests that MHC scents might not be easily recognized under naturalistic conditions, despite the ability of mice to discriminate MHC scents in laboratory tests in which background variation is suppressed [7, 8, 40]. Although MUPs are produced in large quantity and are very resistant to degradation, providing a persistent signal that can be left in the environment as well as for direct communication [13], the short peptide ligands bound by MHC [8] lack the structural features for proteolytic resistance and are therefore highly susceptible to both endoproteolytic and exoproteolytic attack. This would limit their reliability and persistence in scent signals. Although this does not mean that MHC has no role in communication through scent, this might be more limited than previously assumed. MUP-like orthologs are likely to be present in other species that use scent communication because proteins can provide direct information on genetic identity, but as yet there has been little investigation of alternative polymorphic markers beyond MHC. Initial studies indicate that MUP-like lipocalins are expressed by many rodents with a high degree of species specificity and diversity of expression [17]. However, species that rely on visual, acoustic, or other forms of communication might encode genetic identity information through alternative polymorphic mechanisms.

The second general implication is that the ability to recognize kin as a mechanism to avoid inbreeding might be more variable across species than previously considered. Mate-selection mechanisms to avoid the deleterious consequences of inbreeding are only likely to be important where kin encounter each other as adults [2]. Because house mice often live at high density in family-based social groups to exploit locally abundant resources [25], the need to recognize kin to avoid inbreeding will be relatively strong. However, the extent of polymorphism in MUP-like urinary proteins varies considerably between Mus species [17]. Notably, MUPs are not polymorphic in Mus macedonicus [16], a closely related grassland species that lives at much lower densities where individuals are much more widely dispersed than house mice, and thus the need to recognize kin to avoid inbreeding is likely to be considerably less. The ability to avoid inbreeding through kin recognition between adults might be restricted to those species that have evolved specific polymorphic communication signals to achieve this rather than a general ability across vertebrates. Some species might also rely on familiarity of signals learnt during rearing rather than self-referent phenotype matching if relatives reliably interact in the absence of nonkin [41]. The understanding of these issues promises to provide considerable insight into the importance of kin recognition for the avoidance of inbreeding and maintenance of genetic heterozygosity in different social systems. It is timely to extend the scope of research into such polymorphic signaling systems beyond the MHC.


We thank Richard Humphries, John Waters, Felicity Fair, Linda Burgess, and Steve Ramm for practical help and Jane Rees for comments on an earlier draft of the manuscript. The work was supported by the Biotechnology and Biological Science Research Council (grant S19816), the Natural Environment Research Council (studentship NER/S/A/2004/12816), and the Wellcome Trust (064691 and 071257/Z/03).

Supplemental Data

Document S1. Experimental Procedures, Four Figures, and Four Tables:


1. Brown J.L., Eklund A. Kin recognition and the major histocompatibility complex - an integrative review. Am. Nat. 1994;143:435–461.
2. Pusey A., Wolf M. Inbreeding avoidance in animals. Trends Ecol. Evol. 1996;11:201–206. [PubMed]
3. Yamazaki K., Boyse E.A., Mike V., Thaler H.T., Mathieson B.J., Abbott J., Boyse J., Zayas Z.A., Thomas L. Control of mating preferences in mice by genes in the major histocompatibility complex. J. Exp. Med. 1976;144:1324–1335. [PMC free article] [PubMed]
4. Potts W.K., Manning C.J., Wakeland E.K. Mating patterns in seminatural populations of mice influenced By MHC genotype. Nature. 1991;352:619–621. [PubMed]
5. Jordan W.C., Bruford M.W. New perspectives on mate choice and the MHC. Heredity. 1998;81:239–245. [PubMed]
6. Penn D.J. The scent of genetic compatibility: Sexual selection and the major histocompatibility complex. Ethology. 2002;108:1–21.
7. Carroll L.S., Penn D.J., Potts W.K. Discrimination of MHC-derived odors by untrained mice is consistent with divergence in peptide-binding region residues. Proc. Natl. Acad. Sci. USA. 2002;99:2187–2192. [PMC free article] [PubMed]
8. Brennan P.A., Kendrick K.M. Mammalian social odours: Attraction and individual recognition. Philos. Trans. R. Soc. B Biol. Sci. 2006;361:2061–2078. [PMC free article] [PubMed]
9. Slev P.R., Nelson A.C., Potts W.K. Sensory neurons with MHC-like peptide binding properties: disease consequences. Curr. Opin. Immunol. 2006;18:608–616. [PubMed]
10. Thom M.D., Stockley P., Beynon R.J., Hurst J.L. Scent, mate choice and genetic heterozygosity. In: Hurst J.L., Beynon R.J., Roberts S.C., Wyatt T., Yord N., editors. Volume 11. Springer; New York: 2007. pp. 291–301. (Chemical Signals in Vertebrates).
11. Piertney S.B., Oliver M.K. The evolutionary ecology of the major histocompatibility complex. Heredity. 2006;96:7–21. [PubMed]
12. Hurst J.L., Payne C.E., Nevison C.M., Marie A.D., Humphries R.E., Robertson D.H., Cavaggioni A., Beynon R.J. Individual recognition in mice mediated by major urinary proteins. Nature. 2001;414:631–634. [PubMed]
13. Hurst J.L., Beynon R.J. Scent wars: The chemobiology of competitive signalling in mice. Bioessays. 2004;26:1288–1298. [PubMed]
14. Beynon R.J., Hurst J.L. Urinary proteins and the modulation of chemical scents in mice and rats. Peptides. 2004;25:1553–1563. [PubMed]
15. Cheetham S.A., Thom M.D., Jury F., Ollier W.E.R., Beynon R.J., Hurst J.L. The genetic basis of individual-recognition signals in the mouse. Curr. Biol. 2007;17:1771–1777. [PubMed]
16. Robertson D.H.L., Hurst J.L., Searle J.B., Gunduz I., Beynon R.J. Characterization and comparison of major urinary proteins from the house mouse, Mus musculus domesticus, and the aboriginal mouse, Mus macedonicus. J. Chem. Ecol. 2007;33:613–630. [PubMed]
17. Beynon R.J., Turton M.J., Robertson D.H.L., Armstrong S.D., MacNicoll A., Humphries R.E., Hurst J.L. Urinary lipocalins in Rodenta: Is there a generic model? In: Hurst J.L., Beynon R.J., Roberts S.C., Wyatt T., editors. Volume 11. Springer; New York: 2007. pp. 37–50. (Chemical Signals in Vertebrates).
18. Ferris S.D., Sage R.D., Wilson A.C. Evidence from mtDNA sequences that common laboratory strains of inbred mice are descended from a single female. Nature. 1982;295:163–165. [PubMed]
19. Yang H., Bell T.A., Churchill G.A., Pardo-Manuel de Villena F. On the subspecific origin of the laboratory mouse. Nat. Genet. 2007;39:1100–1107. [PubMed]
20. Manning C.J., Potts W.K., Wakeland E.K., Dewsbury D.A. What's wrong with MHC mate choice experiments? In: Doty R.L., Muller-Schwarze D., editors. Chemical Signals in Vertebrates. Plenum; New York: 1992. pp. 229–235.
21. Penn D., Potts W. MHC-disassortative mating preferences reversed by cross-fostering. Proc. R. Soc. Lond. Ser. B Biol. Sci. 1998;265:1299–1306. [PMC free article] [PubMed]
22. Roberts S.C., Gosling L.M. Genetic similarity and quality interact in mate choice decisions by female mice. Nat. Genet. 2003;35:103–106. [PubMed]
23. Robertson D.H., Cox K.A., Gaskell S.J., Evershed R.P., Beynon R.J. Molecular heterogeneity in the Major Urinary Proteins of the house mouse Mus musculus. Biochem. J. 1996;316:265–272. [PMC free article] [PubMed]
24. Hurst J.L., Thom M.D., Nevison C.M., Humphries R.E., Beynon R.J. MHC odours are not required or sufficient for recognition of individual scent owners. Proc. R. Soc. B Biol. Sci. 2005;272:715–724. [PMC free article] [PubMed]
25. Barnard C.J., Hurst J.L., Aldhous P. Of mice and kin - the functional significance of kin bias in social behavior. Biological Reviews of the Cambridge Philosophical Society. 1991;66:379–430. [PubMed]
26. Dean M.D., Ardlie K.G., Nachman M.W. The frequency of multiple paternity suggests that sperm competition is common in house mice (Mus domesticus) Mol. Ecol. 2006;15:4141–4151. [PMC free article] [PubMed]
27. Keller L.F., Waller D.M. Inbreeding effects in wild populations. Trends Ecol. Evol. 2002;17:230–241.
28. Zeh J.A., Zeh D.W. The evolution of polyandry. 2. Post-copulatory defences against genetic incompatibility. Proc. R. Soc. Lond. Ser. B Biol. Sci. 1997;264:69–75.
29. Lehmann L., Perrin N. Inbreeding avoidance through kin recognition: Choosy females boost male dispersal. Am. Nat. 2003;162:638–652. [PubMed]
30. Kokko H., Ots I. When not to avoid inbreeding. Evolution Int. J. Org. Evolution. 2006;60:467–475. [PubMed]
31. Pocock M.J.O., Hauffe H.C., Searle J.B. Dispersal in house mice. Biological Journal of the Linnean Society. 2005;84:565–583.
32. Tregenza T., Wedell N. Genetic compatibility, mate choice and patterns of parentage: Invited review. Mol. Ecol. 2000;9:1013–1027. [PubMed]
33. Thom M.D., Hurst J.L. Individual recognition by scent. Ann. Zool. Fenn. 2004;41:765–787.
34. Potts W.K., Manning C.J., Wakeland E.K. The role of infectious disease, inbreeding and mating preferences in maintaining MHC genetic diversity: an experimental test. Philos. Trans. R. Soc. Lond. B Biol. Sci. 1994;346:369–378. [PubMed]
35. Hughes A.L., Hughes M.K. Natural selection on the peptide-binding regions of major histocompatibility complex-molecules. Immunogenetics. 1995;42:233–243. [PubMed]
36. Pomiankowski A., Pagel M. Sexual selection and MHC genes. Nature. 1992;356:293–294. [PubMed]
37. Beynon R.J., Veggerby C., Payne C.E., Robertson D.H., Gaskell S.J., Humphries R.E., Hurst J.L. Polymorphism in major urinary proteins: Molecular heterogeneity in a wild mouse population. J. Chem. Ecol. 2002;28:1429–1446. [PubMed]
38. Armstrong S.D., Robertson D.H.L., Cheetham S.A., Hurst J.L., Beynon R.J. Structural and functional differences in isoforms of mouse major urinary proteins: A male-specific protein that preferentially binds a male pheromone. Biochem. J. 2005;391:343–350. [PMC free article] [PubMed]
39. Novotny M.V. Pheromones, binding proteins and receptor responses in rodents. Biochem. Soc. Trans. 2003;31:117–122. [PubMed]
40. Schaefer M.L., Young D.A., Restrepo D. Olfactory fingerprints for major histocompatibility complex-determined body odors. J. Neurosci. 2001;21:2481–2487. [PubMed]
41. Mateo J.M. Recognition systems and biological organization: The perception component of social recognition. Ann. Zool. Fenn. 2004;41:729–745.

Save items

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


  • Cited in Books
    Cited in Books
    NCBI Bookshelf books that cite the current articles.
  • MedGen
    Related information in MedGen
  • PubMed
    PubMed citations for these articles
  • Substance
    PubChem chemical substance records that cite the current articles. These references are taken from those provided on submitted PubChem chemical substance records.

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...