Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small “many analyst” study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

The “environmental heterogeneity hypothesis” predicts that variability in resources promotes species coexistence, but few experiments support this hypothesis in plant communities. A previous 15‐yr test of this hypothesis in a prairie restoration experiment demonstrated a weak effect of manipulated soil resource heterogeneity on plant diversity. This response was attributed to a transient increase in richness following a post‐restoration supplemental propagule addition, occasionally higher diversity under nutrient enrichment, and reduced cover of a dominant species in a subset of soil treatments. Here, we report community dynamics under continuous propagule addition in the same experiment, corresponding to 16–20 yr of restoration, in response to altered availability and heterogeneity of soil resources. We also quantified traits of newly added species to determine if heterogeneity increases the amount and variety of niches available for new species to exploit. The heterogeneous treatment contained a factorial combination of altered nutrient availability and soil depth; control plots had no manipulations. Total diversity and richness were higher in the heterogeneous treatment during this 5‐yr study due to higher cover, diversity, and richness of previously established forbs, particularly in the N‐enriched subplots. All new species added to the experiment exhibited unique trait spaces, but there was no evidence that heterogeneous plots contained a greater variety of new species representing a wider range of trait spaces relative to the control treatment. The richness and cover of new species was higher in N‐enriched soil, but the magnitude of this response was small. Communities assembling under long‐term N addition were dominated by different species among subplots receiving added N, leading to greater dispersion of communities among the heterogeneous relative to control plots. Contrary to the deterministic mechanism by which heterogeneity was expected to increase diversity (greater variability in resources for new species to exploit), higher diversity in the heterogeneous plots resulted from destabilization of formerly grass‐dominated communities in N‐enriched subplots. While we do not advocate increasing available soil N at large scales, we conclude that the positive effect of environmental heterogeneity on diversity can take decades to materialize and depend on development of stochastic processes in communities with strong establishment limitation.

Plant community assembly outcomes can be contingent upon establishment year (year effects) due to variations in the environment. Stochastic events such as interannual variability in climate, particularly in the first year of community assembly, contribute to unpredictable community outcomes over the short term, but less is known about whether year effects produce transient or persistent states on a decadal timescale. To test for short‐term (5‐year) and persistent (decadal) effects of establishment year climate on community assembly outcomes, we restored prairie in an agricultural field using the same methods in four different years (2010, 2012, 2014, and 2016) that captured a wide range of initial (planting) year climate conditions. Species composition was measured for 5 years in all four restored prairies and for 9 and 11 years in the two oldest restored prairies established under average precipitation and extreme drought conditions. The composition of the four assembled communities showed large and significant differences in the first year of restoration, followed by dynamic change over time along a similar trajectory due to a temporary flush of annual volunteer species. Sown perennial species eventually came to dominate all communities, but communities remained distinct from each other in year five. Precipitation in June and July of the establishment year explained short‐term coarse community metrics (i.e., species richness and grass/forb cover), with wet establishment years resulting in a higher cover of grasses and dry establishment years resulting in a higher cover of forbs in restored communities. Short‐term differences in community composition, species richness, and grass/forb cover in restorations established under average precipitation and drought conditions persisted for 9–11 years, with low interannual variability in the composition of each prairie over the long term, indicating persistently different states on a decadal timescale. Thus, year effects resulting from stochastic variation in climate can have decadal effects on community assembly outcomes.