This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
3 November 2016 in Journal of Youth and Adolescence
Life satisfaction is an important indicator of successful development. However, adolescents’ life satisfaction tends to be relatively unsteady, and environmental influences play a critical role in shaping life satisfaction among adolescents in the transition to young adulthood. Given the paramount importance that education plays in adolescents’ lives, adolescents’ life satisfaction may vary as a function of school-related stress experience. At the same time, coping resources may help reduce adverse effects of stress on life satisfaction. With this in mind, we examined whether, and to what extent, perceived stress in education and general self-efficacy (a resource that facilitates coping) affect the life satisfaction of adolescents in transition to young adulthood. We distinguished between baseline levels of stress and self-efficacy and within-person change in stress and self-efficacy to determine whether life satisfaction is sensitive to fluctuations in stress and self-efficacy when person-specific levels of stress and self-efficacy are taken into account. Estimating growth curve models on data from a panel study on the life trajectories of compulsory-school leavers (n = 5126, 55.3 % female), we found that baseline levels of stress and self-efficacy, as well as within-person change in stress and self-efficacy, affected adolescents’ life satisfaction. Moreover, our results showed that baseline self-efficacy mitigated the negative effect of baseline stress on life satisfaction. These findings improve our understanding of two major psychological determinants of adolescents’ life satisfaction and extend our knowledge of life satisfaction trajectories during the transition to young adulthood.
Expectations about the future direct effort in goal-oriented action and may influence a range of life course outcomes, including educational attainment. Here we investigate whether such expectations are implicated in the dynamics underlying the persistence of educational advantage across family generations, and whether such dynamics have changed in recent decades in view of historical change. Focusing on the role of domain-specific (educational) and general (optimism and control) expectations, we examine parallels across parent-child cohorts in (1) the relationships between parental socioeconomic status and children’s future expectations and (2) the associations between children’s future expectations and their academic achievement. We estimate structural equation models using data from the prospective multigenerational Youth Development Study (N = 422 three-generation triads [G1-G2-G3]; G1 mean age in 1988 = 41.0 years, G2 mean age in 1989 = 14.7 years, G3 mean age in 2011 = 15.8 years; G2 white in 1989 = 66.4%, G3 white in 2011 = 64.4%; G1 mean annual household income, converted to 2008 equivalents = $41,687, G2 mean annual household income in 2008 dollars = $42,962; G1 mode of educational attainment = high school, G2 mode of educational attainment = some college). We find intergenerational similarity in the relationships between parental educational attainment and children’s future expectations. Children’s educational expectations strongly predicted their academic achievement in the second generation, but not in the third generation. With educational expansion, the more recent cohort had higher educational expectations that were less strongly related to achievement. Overall, the findings reveal dynamics underlying the persistence of educational success across generations. The role of future expectations in this intergenerational process varies across historical time, confirming a central conclusion of life-span developmental psychology and life-course sociological research that individual functioning is influenced by sociocultural contexts.