All of the following are examples of prezygotic genetic isolating mechanisms except:

Uniform Selection (Mutation-Order Speciation)

Reproductive isolation might also evolve during a process of mutation-order speciation, defined as the evolution of reproductive isolation by the fixation of different advantageous mutations in separate populations experiencing similar selection pressures, that is, uniform selection. In essence, different populations find different genetic solutions to the same selective problem. In turn, the different genetic solutions (i.e., mutations) are incompatible with one another, causing reproductive isolation. During ecological speciation, different alleles are favored between two populations. By contrast, during mutation-order speciation, the same alleles are favored in both populations, but divergence occurs anyway because, by chance, the populations do not acquire the same mutations or fix them in the same order. Divergence is therefore stochastic, but the process involves selection, and, thus, is distinct from random genetic drift. Selection can be ecologically based under mutation-order speciation, but ecology does not favor divergence as such, and an association between ecological divergence and reproductive isolation is not expected. How might mutation-order speciation arise? Sexual selection may cause mutation-order speciation if reproductive isolation evolves by the fixation of alternative advantageous mutations in different populations living in similar ecological environments.

Genetic Analysis of Reproductive Isolation: Principles

Reproductive isolation is unique in evolution because it is not a trait possessed by members of a single species, but a composite character that is the joint property of a pair of species. A single species can be reproductively isolated only with respect to another. Moreover, by its very nature, reproductive isolation is a trait that almost always involves epistatic interaction between alleles – but alleles occurring in different species. Hybrid inviability, for example, results from genes that produce normal viability in members of their own species but are lethal when interacting with alien genes in hybrids. Similarly, sexual isolation is caused when females evolved to prefer traits of conspecific males encounter different traits in other species. This composite and epistatic nature of reproductive isolation guarantees that speciation will not only show emergent genetic and phenotypic properties not seen in studies of a single species (e.g., Haldanes rule; see below), but also that mathematical theories of speciation will be different – and perhaps more complicated – than models of evolution in single lineages. While genetic analysis of reproductive isolation has occurred since the mid-1930s, mathematical theories of speciation are only now beginning to appear.

There are several reasons for studying the genetic basis of speciation. First, just as with a trait that evolves within a lineage, one wants to know whether a reproductive isolating mechanism has a ‘simple’ genetic basis (i.e., involves only a few genes of large effect) or is based on the accumulation of many genes. The number of genes involved may, in turn, allow inferences about the evolutionary process producing reproductive isolation. For example, if the difference in plumage color between males of two sexually dimorphic bird species is due to many genes of small effect, one may posit that these differences arose by sexual selection during which the male trait evolved step-by-step in concert with the female preference for that trait.

Similarly, the pattern of genetic differences causing reproductive isolation may give clues to the underlying evolutionary processes. It has been found, for example, that there are often many more genes causing hybrid male than female sterility between closely related species of Drosophila. This has led to the idea that hybrid sterility may result from sexual selection acting in isolated populations. Such selection, based on female choice, may cause more evolutionary change in males than in females, leading to the preferential sterility of male hybrids as an accidental outcome. Finally, genetic analysis can help localize small sections of chromosomes containing genes causing reproductive isolation, a necessary prelude to cloning and sequencing these genes. Such molecular work is essential for understanding the developmental basis of reproductive isolation, including the question of how a gene that works normally within a species causes deleterious effects in hybrids. At this writing we understand the developmental basis of only one case of reproductive isolation: the formation of lethal melanomas in hybrids between the swordfish and platyfish. This hybrid lethality is based on an oncogene in one species that is normally suppressed by another gene in the same species; the absence of suppression in hybrids causes the appearance of tumors.

Ideally, a study of the genetics of speciation should involve only reproductive isolating mechanisms that evolved up to the point at which gene exchange between populations was first reduced to zero, for it is at that point that speciation is complete. Because of divergent evolution, however, reproductive isolation continues to accumulate even after species cannot exchange genes, but such isolation is incidental to speciation. A proper study of speciation thus requires identifying the isolating mechanisms leading up to complete isolation (there may, of course, be more than one). This is not easy, as it requires that one must find either incipient species that have not yet evolved complete reproductive isolation, or species in which gene flow is prevented by only a single form of reproductive isolation. This has been possible in some cases, as with polyploidy in plants (see below), but in no group of animals or plants have there been systematic attempts to determine which forms of reproductive isolation are the first to evolve. Instead, there are only tentative conclusions based on general impressions. It has been suggested, for example, that sexual isolation is the most important factor causing speciation in birds, as closely related species do not hybridize in the wild but will produce fertile hybrids when forcibly crossed in the laboratory. Such suppositions are intriguing, but neglect possible ecological isolation, and must be buttressed by systematic analysis of populations at different stages of evolutionary divergence.

Prezygotic Isolation

Reproductive isolation represents a breakdown in the ability to reproduce successfully with sexual partners of another type of organism, and speciation requires a build up of reproductive isolation between diverging types of organism until gene flow is sufficiently rare or ineffective that the entities are considered ‘good species.’ Traditionally this was thought to require complete or near complete cessation of gene flow, though increasingly absolute reproductive isolation is thought to be too stringent a criterion (Mallet, 1995; Wu, 2001). Factors which influence prezygotic isolation are those that come into play before gametes of the different types meet and form zygotes. After this point postzygotic isolation occurs, and this simple classification of categories of reproductive isolation based on pre- and post-gametic fusion has been widely adopted since Dobzhansky originally categorized major factors influencing the origin of species into various ‘reproductive isolating mechanisms’ (Dobzhansky, 1937). However, it is important to appreciate that all factors influencing reproductive isolation act in combination. If cross-matings between males and females of different types are half as likely as within types we say their isolation index (I) is 0.5. If the viability of their offspring is also around 50% these are equally effective barriers to gene flow, and acting together will produce a combined I of 0.75, though prezygotic isolation will have made a greater contribution to the overall isolation only because it occurs first (Coyne and Orr, 2004).

What are Speciation Genes?

Studies of reproductive isolation have attempted to isolate the genes responsible for reproductive isolation. However, the very definition of what a speciation gene is makes it difficult to agree how many speciation genes have been found, and whether they reveal patterns of speciation (Rieseberg and Blackman, 2010; Presgraves, 2010a,b). Under the view that speciation is the evolution of reproductive isolation, it is clear that genes whose alleles contribute to some form of reproductive isolation can be considered a speciation gene (Nosil and Schluter, 2011). On one hand, there are genes contributing to DBM genetic incompatibilities, and therefore to the evolution of intrinsic reproductive isolation (hybrid sterility and inviability). On the other hand there are genes responsible for local adaptation and the evolution of extrinsic reproductive isolation (immigrant inviability and extrinsic postzygotic isolation), and those responsible for gametic recognition, for male–female interactions, and mate choice. In this sense, the list of speciation genes that have been discovered is perhaps larger than anticipated; yet they still have not clearly revealed whether there are special categories of speciation genes.

Questions about the contribution of a particular gene to speciation are similar to those applied to measurements of reproductive isolation. It would be important to know when the speciation gene arose, and whether or not speciation genes create full or weak reproductive isolation (Nosil and Schluter, 2011; Coyne and Orr, 2004). In relative terms, a speciation gene causing strong reproductive isolation might see its effects reduced to a very small magnitude if it arose very late in the speciation process. In a similar fashion, a gene whose expression is late during the life cycle of the organism (e.g., hybrid sterility), might also contribute relatively little to total reproductive isolation if genes expressed earlier during development (e.g., genes responsible for seed germination in plants) already produced high levels of isolation between populations.

What are species? The biological species concept

Disagreement as to what precisely constitutes a species is to be expected, given that the concept serves so many functions (Vane-Wright, 1992). We may be interested in classification as such, or in the evolutionary implications of species; in the theory of species, or in simply how to recognize them; or in their reproductive, physiological, or husbandry status.

Most non-specialists probably have some vague idea that species are defined by not interbreeding with each other; usually, that hybrids between different species are sterile, or that they are incapable of hybridizing at all. Such an impression ultimately derives from the definition by Mayr (1940), whereby species are “groups of actually or potentially interbreeding natural populations which are reproductively isolated from other such groups” (the Biological Species Concept). Mayr never actually said that species can't breed with each other, indeed he denied that that this was in any way a necessary part of reproductive isolation; he merely said that, under natural conditions, they don't.

Reproductive isolation, in some form, stands at the basis of what a species is. Having said this, it must be admitted that it is no longer possible to follow Mayr's concept as definitive. In a recent book (Groves, 2001, see especially Chapter 3) I sketched the main reasons why this is so:

It offers no guidance for the allocation of allopatric populations.

Many distinct species actually do breed with each other under natural conditions, but manage to remain distinct.

The interrelationships of organisms under natural conditions are often (usually?) unknown.

Many species do not reproduce sexually anyway.

Speciation and Behavior

Finally, the formation of species (called speciation) provides a powerful tool for asking questions about the history of traits, including behavioral traits. So far, this review of evolution has addressed changes in gene frequency and the manner in which those changes happen. How can such genetic shifts within a population of animals result in the biological diversity we see today? Something else has to happen: the flow of genes within that population must be interrupted. This can occur when a population is subdivided and parts of that population are isolated from each other. Eventually, if environments of the subdivided groups differ, natural selection will favor different traits in the two new populations. As time passes, differences accumulate, and the two populations will no longer be able to interbreed were they to have that opportunity. For instance, foxes, coyotes, wolves, and domestic dogs all evolved from the same ancestor, with foxes diverging earliest in evolutionary time. Differences have accumulated so that now foxes cannot successfully interbreed with any of the other species. However, coyotes, wolves, and domestic dogs split more recently and remain so similar that hybrids are common. By the way, complete isolation is not necessary for species formation; a small amount of gene flow does not counteract the accumulation of differences.

Such reproductive isolation can result from a variety of causes:

Geographic barriers. If a population is subdivided by the emergence of a mountain range, river, or other inhospitable habitat, animals on one side of the barrier will be unable to breed with animals on the other side. The same effect occurs if part of the population moves away.

Resource shifts. For animals that live and reproduce on a resource, the ability to colonize new resources decreases the likelihood that they will encounter or mate with individuals in the parent population.

Mate choice. If females diverge in their preferences for male characteristics, for instance, and if that divergence has a genetic basis, then eventually there will be two distinct gene pools, each sporting one or the other preferred male trait.

Genetic change. Mutations that prevent proper meiosis can produce individuals that cannot mate with other members of the population. This is thought to be the origin of about 4% of plant species.

The Genetics of Reinforcement

The dilemma of reproductive isolation evolving in sympatry has also sparked theoretical work investigating the genetic basis of reinforcement. Specifically, it has been shown that the genetic architecture of the reproductive isolating mechanism can influence the success of reinforcing selection. If a mutant allele increases RI when it occurs in either or both of the two sympatric species, then reinforcement is more likely to be successful. This is referred to as a one-allele mechanism (Felsenstein, 1981). For example, an allele that increases ‘choosiness’ or reduces dispersal could cause both sympatric species to mate more frequently with conspecifics than heterospecifics. Despite much hypothesizing about the types of traits that could be one-allele mechanisms of reinforcement, it has only actually been identified once. Drosophila pseudoobscura individuals that co-occur with Drosophila persimilis show a greater reluctance to hybridize than individuals from allopatric populations. The sympatric allele at one locus, Coy-2, that increases assortative mating in D. pseudoobscura, has also been shown to increase assortative mating when it is occurring in D. persimilis as well (Ortiz-Barrientos and Noor, 2005). In other words, the same allele in either species will cause a decrease in hybridization. The one-allele mechanism overcomes the problems associated with gene flow because recombination of the derived assortative mating allele into the sympatric species will increase reproductive isolation instead of decrease it.

Besides some work on genetic architecture of trait variation caused by reinforcement, little is known about the genetics of reinforcement. In fact, Phlox is the only system for which the genes involved in reinforcement have been identified. Flower color variation in P. drummondii is caused by expression variation in two genes known to be involved in pigment production in flowering plants. Yet even in this case, we know little about the evolutionary history of these genes. Did a single mutation arise once and spread or are there multiple mutations causing the same changes in flower color? Did the alleles causing flower color variation arise under selection or were they segregating in the population prior to the two Phlox species hybridizing? Clearly, much more research is needed before we can determine if there are particular genetic or genomic patterns associated with reinforcement.

2.3 Differentiation: Selection and Chance

Both the development of reproductive isolation and the differentiation of other characters identifying organisms as members of a species could be influenced by selection or by genetic drift. In the former case, adaptation to a particular set of environmental conditions drives speciation. In the latter case, chance events, such as the genetic diversity present in individuals that become geographically isolated, or that happens to be included when a polyploid individual is formed, influence the results of speciation. The relative influence of these two forces has been the subject of debate; like many polarized debates in biology, most cases probably fall along the continuum between the two ends.

Evolutionary Characterization of Speciation Genes

Recognizing that the evolution of RI is a temporally dynamic process that begins prior to and continues after speciation, the definition advanced above also requires that the gene ‘contribute to the splitting of two lineages.’ Genetic changes that contribute to contemporary barriers to reproduction between fully isolated species may have accumulated long after speciation was already complete. Likewise, the relative importance of different barriers to gene flow, and consequently the impact of particular alleles on total RI, may change over time. Therefore, it has been argued that speciation genes must meet two additional evolutionary criteria (Nosil and Schluter, 2011). First, divergence at the locus must have occurred before speciation was complete. Second, the gene should have had a measurable effect on RI at the time it diverged. Obtaining empirical data addressing the first criterion can be straightforward, but demonstrating a gene meets the second criterion is potentially far more challenging.

The divergence time criterion is automatically met for genes that contribute to barriers between incipient species. For species pairs that are already completely isolated, divergence time can be assessed with gene trees or genealogies (Nosil and Schluter, 2011). If divergence at a putative speciation gene occurred prior to full cessation of gene flow, then its genealogy should be discordant with genealogies of unlinked neutral loci, which are more likely to show shallower coalescence and greater evidence of gene flow. Such tests can be problematic, however, in that they may yield false negatives depending on the evolutionary process driving divergence (Lessios, 2011). For instance, RI between species often arises as a by-product of accelerated sequence evolution driven by arms races mediated by sexual or genomic conflict within species that continue well after speciation is complete. Thus, a pattern of coalescence within species, rather than a pattern of coalescence prior to the timing of species divergence, may be expected and has been observed for some speciation genes evolving in this manner (Palumbi, 2009).

As for the second criterion, it is unclear whether a speciation gene’s effect on RI relative to all other current barriers at the time of its divergence can be rigorously estimated for any species. The absolute effect of allelic divergence at a speciation gene on the strength of a contemporary RI barrier may be readily estimated by the methods discussed above. Moreover, a speciation gene’s relative contribution to contemporary RI as a whole may also be estimated to the extent that the full genetic architecture for the barrier trait and the relative strength of that barrier relative to other barrier traits on RI are known. All these parameters are important because isolating barriers, to a large extent, act sequentially to prevent successful fertilization or to impair hybrid fitness, and hence it is possible that loci with major effects on later acting barriers (e.g., hybrid dysfunction) may only have minor contributions to overall RI.

By extension then, estimating a speciation gene’s contribution to historical levels of RI would require not only knowledge of the temporal dynamics of evolution at a given locus (and interacting loci if RI is caused by epistatic incompatibilities) in one or both lineages, but also the past history of all other loci contributing to the cessation of gene flow. Thus, meeting this criteria may only be possible in rare systems where speciation in action can be followed over observable time scales. Alternatively, in species pairs isolated by few barriers with tractable genetics, it is conceivable that the historical series of genotypes at speciation loci could be reconstructed if the order of substitutions causing RI were inferable from genes trees or observable in ancient DNA time series. Estimates of absolute and relative contributions to contemporary RI may help determine the bounds to paths historically possible in other systems. However, given the pragmatic hurdles to determining historical dynamics (irrespective of any complexity introduced by gene×environment effects), it seems sufficient to demonstrate a speciation gene affects contemporary isolation and acknowledge the caveat that the specific contribution to total RI at the time of divergence is unknown.

Barriers to Gene Exchange

Given that the key question in speciation is how populations of organisms diverge into genetically distinguishable entities, it is important to understand and categorize the different types of gene flow barriers that can evolve to generate new species. Two pioneers in this area were Poulton (1908) and Dobzhansky (1937) who divided RI into two different major categories of prezygotic and postzygotic isolation depending upon whether the barrier to gene exchange occurred prior to fertilization (zygote creation) or after (see Table 1 for summary of different types of pre- and post-zygotic barriers to gene exchange and examples).

Table 1. Barriers to gene flow between populations causing reproductive isolation (RI) and examples of each barrier