This phenomenon, called cascade reinforcement, has been observed in insects and amphibians, but not in fish, said University of Illinois animal biology professor Rebecca Fuller, who led the new analysis.
Reinforcement can occur when two species living in the same environment mate and produce inferior, hybrid offspring. Fish that mate with other species have fewer offspring that live to adulthood, while fish that mate only within their own species are likely to have more descendants in future generations.
A possible consequence of this process of natural selection occurs when females that avoid mating with other species take their choosiness one step further - also shunning certain populations of potential mates from their own species, Ms Fuller said.
"A process that benefits the fish in one context - avoiding mating with another species - bleeds over into a different context, which may or may not be beneficial," she said.
The new study looked at mate preferences in two species: rainwater killifish (Lucania parva) and bluefin killifish (Lucania goodei). These small, adaptable fish sometimes share a habitat and can mate and produce offspring. L. parva live in marine, freshwater and brackish habitats, while L. goodei prefer fresh water but can tolerate brackish waters.
Previous studies have shown that when L. parva and L. goodei share a habitat, males and females develop a strong preference for mates of their own species. The new analysis revealed that L. parva females that live in proximity to L. goodei also tend to shun males of their own species if the males belong to a different population than their own.
"We've known for a good 20 years that there's selection for animals to have particularly strong preferences for members of their own species when they co-occur with another species that they can hybridize with," Ms Fuller said.
"We're now starting to realize that this can also sometimes create a correlated response where they also prefer members of their own species that are from their own population. And this may lead to the early stages of speciation. It may start the speciation process."
More research must be done to establish that such preferences can lead to the advent of new species, but the findings add to the evidence that "biodiversity begets biodiversity," Ms Fuller said.
The findings are reported in The American Naturalist.
