The use of Project Gutenberg (PG) as a text corpus has been extremely popular in statistical analysis of language for more than 25 years. However, in contrast to other major linguistic datasets of similar importance, no consensual full version of PG exists to date. In fact, most PG studies so far either consider only a small number of manually selected books, leading to potential biased subsets, or employ vastly different pre-processing strategies (often specified in insufficient details), raising concerns regarding the reproducibility of published results. In order to address these shortcomings, here we present the Standardized Project Gutenberg Corpus (SPGC), an open science approach to a curated version of the complete PG data containing more than 50,000 books and more than word-tokens. Using different sources of annotated metadata, we not only provide a broad characterization of the content of PG, but also show different examples highlighting the potential of SPGC for investigating language variability across time, subjects, and authors. We publish our methodology in detail, the code to download and process the data, as well as the obtained corpus itself on three different levels of granularity (raw text, timeseries of word tokens, and counts of words). In this way, we provide a reproducible, pre-processed, full-size version of Project Gutenberg as a new scientific resource for corpus linguistics, natural language processing, and information retrieval.
21 May 2018 in Proceedings of the National Academy of Sciences
Significance: Cells can change their phenotype from epithelial to mesenchymal during development and in cancer progression, where this transition is often associated with metastasis and poor disease prognosis. Here we show this process involves the transit through a multitude of metastable hybrid phenotypes in a way that is similar to the driven dynamics of disordered materials. Our method shows that highly aggressive hybrid epithelial/mesenchymal cell phenotypes are located in metastable regions that can easily switch under external and internal perturbations. We propose a general mapping strategy that can be used for other pathways, providing a useful tool to visualize the ever increasing number of gene expression data obtained from single cells and tissues.
Despite being a paradigm of quantitative linguistics, Zipf’s law for words suffers from three main problems: its formulation is ambiguous, its validity has not been tested rigorously from a statistical point of view, and it has not been confronted to a representatively large number of texts. So, we can summarize the current support of Zipf’s law in texts as anecdotic. We try to solve these issues by studying three different versions of Zipf’s law and fitting them to all available English texts in the Project Gutenberg database (consisting of more than 30 000 texts). To do so we use state-of-the art tools in fitting and goodness-of-fit tests, carefully tailored to the peculiarities of text statistics. Remarkably, one of the three versions of Zipf’s law, consisting of a pure power-law form in the complementary cumulative distribution function of word frequencies, is able to fit more than 40% of the texts in the database (at the 0.05 significance level), for the whole domain of frequencies (from 1 to the maximum value), and with only one free parameter (the exponent).
Mechanical metamaterial actuators achieve pre-determined input–output operations exploiting architectural features encoded within a single 3D printed element, thus removing the need for assembling different structural components. Despite the rapid progress in the field, there is still a need for efficient strategies to optimize metamaterial design for a variety of functions. We present a computational method for the automatic design of mechanical metamaterial actuators that combines a reinforced Monte Carlo method with discrete element simulations. 3D printing of selected mechanical metamaterial actuators shows that the machine-generated structures can reach high efficiency, exceeding human-designed structures. We also show that it is possible to design efficient actuators by training a deep neural network which is then able to predict the efficiency from the image of a structure and to identify its functional regions. The elementary actuators devised here can be combined to produce metamaterial machines of arbitrary complexity for countless engineering applications.
Quantifying the similarity between symbolic sequences is a traditional problem in information theory which requires comparing the frequencies of symbols in different sequences. In numerous modern applications, ranging from DNA over music to texts, the distribution of symbol frequencies is characterized by heavy-tailed distributions (e.g., Zipf’s law). The large number of low-frequency symbols in these distributions poses major difficulties to the estimation of the similarity between sequences; e.g., they hinder an accurate finite-size estimation of entropies. Here, we show analytically how the systematic (bias) and statistical (fluctuations) errors in these estimations depend on the sample size and on the exponent of the heavy-tailed distribution. Our results are valid for the Shannon entropy (), its corresponding similarity measures (e.g., the Jensen-Shanon divergence), and also for measures based on the generalized entropy of order . For small ’s, including , the errors decay slower than the decay observed in short-tailed distributions. For larger than a critical value , the decay is recovered. We show the practical significance of our results by quantifying the evolution of the English language over the last two centuries using a complete spectrum of measures. We find that frequent words change more slowly than less frequent words and that provides the most robust measure to quantify language change.