Emergence and loss of assortative mating in sympatric speciation Fabiano Ribeiro à , Nestor Caticha Instituto de Fı ´sica, Universidade de Sa ˜o Paulo, CP66318, CEP 05315-970, Sa ˜o Paulo, SP, Brazil article info Article history: Received 12 February 2008 Received in revised form 19 November 2008 Accepted 27 November 2008 Available online 13 December 2008 Keywords: Evolution of assortative mating Reproductive isolation Sympatric speciation Phase transitions abstract We have studied an agent model which presents the emergence of sexual barriers through the onset of assortative mating, a condition that might lead to sympatric speciation. In the model, individuals are characterized by two traits, each determined by a single locus A or B. Heterozygotes on A are penalized by introducing an adaptive difference from homozygotes. Two niches are available. Each A homozygote is adapted to one of the niches. The second trait, called the marker trait has no bearing on the fitness. The model includes mating preferences, which are inherited from the mother and subject to random variations. A parameter controlling recombination probabilities of the two loci is also introduced. We study the phase diagram by means of simulations, in the space of parameters (adaptive difference, carrying capacity, recombination probability). Three phases are found, characterized by (i) assortative mating, (ii) extinction of one of the A alleles and (iii) Hardy–Weinberg like equilibrium. We also make perturbations of these phases to see how robust they are. Assortative mating can be gained or lost with changes that present hysteresis loops, showing the resulting equilibrium to have partial memory of the initial state and that the process of going from a polymorphic panmictic phase to a phase where assortative mating acts as sexual barrier can be described as a first-order transition. & 2009 Published by Elsevier Ltd. 1. Introduction Speciation, the origin of more than one species from an ancestral species is driven by the appearance of a barrier separating groups of the original population. During sympatric speciation, first suggested by Walsh (1864) (see also Ortiz- Barrientos et al., 2004; Tauber and Tauber, 1992; Noor, 1996; Kirkpatrick and Ravigne, 2002) the barrier is not geographic. Maynard Smith (1966) pointed out some situations that can promote sympatric speciation in nature. First among them is the emergence of a sexual preference leading to sexual isolation between some groups of individuals in the population. When that sexual preference discriminates among individuals which vary on a given trait this process is known as assortative mating. Assortative mating can occur, as discussed by Maynard Smith, in an environment with disruptive selection, selection that favors substantially different phenotypes at the expense of intermediate (hybrid) phenotypes. In this paper we study a model which allows for the onset of assortative mating, which in turn may lead to sympatric speciation. Whether assortative mating appears depends on the value of parameters which measure the amount of disruptive selection and the environment’s carrying capacity. If they are changed beyond certain limits there is a transition into the assortative mating state. We are interested in the possibility of reverse speciation and study the consequences of changing these parameters back to the previous region with no speciation. We found hysteresis in the transition to assortative mating, i.e. when the parameters are changed back, the values of the parameters at the transition needed for the onset of assortative mating, are not the same as those needed to achieve reversal of speciation. Hysteresis is a sign of metastability and found in systems which undergo a first-order transition. This is a concept from statistical mechanics which will be explained below. For the terminology of phase transitions see Yeomans (1994). Reverse speciation has been seen in field experiments. Whether hysteresis is found in these cases remains to be seen. Hysteresis loops are only observable if parameters can be changed. The lack of external control to change at will the parameters may hamper experimental observation, but knowing that hysteresis is a possibility may prevent expectations that restoring original conditions may restore the same ecological state. Canadian stickleback fish (Gasterosteus aculeatus) present (see Peichel et al., 2001) loss of assortative mating which in turn leads to the reversal of sympatric speciation. This might be a biological example with properties analogous to those in our model. Benthic and limnetic species, which inhabit five lakes in British Columbia, can be distinguished essentially by shape and size. Their morphologic differences are due to adaptation to the different niches they occupy (Schluter and McPhail, 1992). During the mating period they inhabit the same lake shore region showing a ARTICLE IN PRESS Contents lists available at ScienceDirect journal homepage: www.elsevier.com/locate/yjtbi Journal of Theoretical Biology 0022-5193/$ - see front matter & 2009 Published by Elsevier Ltd. doi:10.1016/j.jtbi.2008.11.027 à Corresponding author. Tel.: +49176 2579 2950. E-mail address: fribeiro@if.usp.br (F. Ribeiro). Journal of Theoretical Biology 258 (2009) 465–477