Despite extensive archaeological research, our knowledge of the human population history of Upper Paleolithic Europe remains limited, primarily due to the scarce availability and poor molecular preservation of fossil remains. As teeth dominate the fossil record and preserve genetic signatures in their morphology, we compiled a large dataset of 450 dentitions dating between ~47 and 7 thousand years ago (ka), outnumbering existing skeletal and paleogenetic datasets. We tested a range of competing demographic scenarios using a coalescent-based machine learning Approximate Bayesian Computation (ABC) framework that we modified for use with phenotypic data. Mostly in agreement with but also challenging some of the hitherto available evidence, we identified a population turnover in western Europe at ~28 ka, isolates in western and eastern refugia between ~28 and 14.7 ka, and bottlenecks during the Last Glacial Maximum. Methodologically, this study marks the pioneering application of ABC to skeletal phenotypes, paving the way for exciting future research avenues.

Sicily, situated at the heart of the Mediterranean Sea, has been a crossroads of people of different origins since the Paleolithic. To gain further insight into the genetic history of this island from a matrilineal viewpoint, we investigated 15 millennia of human mitogenome evolution. A unique Sicilian mitochondrial DNA (mtDNA) dataset, represented by 116 ancient mitogenomes (including two newly sequenced) collected from 16 archeological sites dating from 14,700 to 545 years ago, was compared with a collection of 236 modern mitogenomes covering all districts of the island. By integrating demographic modeling with phylogeographic analyses, we identified a statistically supported genetic discontinuity between the Paleolithic/Late Mesolithic and Early Neolithic periods and two mtDNA lineages (U5b and U8b/K) that specifically mark this transition. The extensive variation and lack of genetic structure among modern mitogenomes suggest the presence of a continuous, maternally inherited gene flow from different regions of Western Eurasia (since the Paleolithic) and Africa (since the Bronze Age).

Inferring past demographic histories is crucial in population genetics, and the amount of complete genomes now available should in principle facilitate this inference. In practice, however, the available inferential methods suffer from severe limitations. Although hundreds complete genomes can be simultaneously analyzed, complex demographic processes can easily exceed computational constraints, and the procedures to evaluate the reliability of the estimates contribute to increase the computational effort. Here we present an Approximate Bayesian Computation (ABC) framework, based on the Random Forest algorithm, to infer complex past population processes using complete genomes. To do this, we propose to summarize the data by the full genomic distribution of the four mutually exclusive categories of segregating sites ( FDSS ), a statistic fast to compute from unphased genome data. We constructed an efficient ABC pipeline and tested how accurately it allows one to recognize the true model among models of increasing complexity, using simulated data and taking into account different sampling strategies in terms of number of individuals analyzed, number and size of the genetic loci considered. We tested the power of the FDSS to be informative about even complex evolutionary histories and compared the results with those obtained summarizing the data through the unfolded Site Frequency Spectrum, thus highlighting for both statistics the experimental conditions maximizing the inferential power. Finally, we analyzed two datasets, testing models (a) on the dispersal of anatomically modern humans out of Africa and (b) the evolutionary relationships of the three species of Orangutan inhabiting Borneo and Sumatra.

The reconstruction of past demographic histories relies on the pattern of genetic variation shown by the sampled populations; this means that an accurate estimation of genotypes is crucial for a reliable inference of past processes. A commonly adopted approach to reconstruct complex demographic dynamics is the Approximate Bayesian Computation (ABC) framework. It exploits coalescent simulations to generate the expected level of variation under different evolutionary scenarios. Demographic inference is then performed by comparing the simulated data with the genotypes called in the sampled individuals. Low sequencing coverage drastically affects the ability to reliably call genotypes, thus making low-coverage data unsuitable for such powerful inferential approaches. Here, we present Low-ABC, a new ABC approach to infer past population processes using low-coverage whole-genome data. Under this framework, both observed and simulated genetic variation are not directly compared using called genotypes, but rather obtained using genotype likelihoods to consider the uncertainty caused by the low sequencing coverage. We first evaluated the inferential power of this procedure in distinguishing among different demographic models and in inferring model parameters under different experimental conditions, including a wide spectrum of sequencing coverage (1x to 30x), number of individuals, number, and size of genetic loci. We showed that the use of genotype likelihoods integrated into an ABC framework provides a reliable inference of past population dynamics, thus making possible the application of model-based inference also for low-coverage data. We then applied Low-ABC to shed light on the relationship between Mesolithic and Early Neolithic European populations.

Light eyes, hair and skins probably evolved several times as Homo sapiens dispersed from Africa. In areas with lower UV radiation, light pigmentation alleles increased in frequency because of their adaptive advantage and of other contingent factors such as migration and drift. However, the tempo and mode of their spread is not known. Phenotypic inference from ancient DNA is complicated, both because these traits are polygenic, and because of low sequence depth. We evaluated the effects of the latter by randomly removing reads in two high-coverage ancient samples, the Paleolithic Ust’-Ishim from Russia and the Mesolithic SF12 from Sweden. We could thus compare three approaches to pigmentation inference, concluding that, for suboptimal levels of coverage (<8x), a probabilistic method estimating genotype likelihoods leads to the most robust predictions. We then applied that protocol to 348 ancient genomes from Eurasia, describing how skin, eye and hair color evolved over the past 45,000 years. The shift towards lighter pigmentations turned out to be all but linear in time and place, and slower than expected, with half of the individuals showing dark or intermediate skin colors well into the Copper and Iron ages. We also observed a peak of light eye pigmentation in Mesolithic times, and an accelerated change during the spread of Neolithic farmers over Western Eurasia, although localized processes of gene flow and admixture, or lack thereof, also played a significant role.