28 October 2022 in Proceedings of the National Academy of Sciences
This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
1 July 2015 in Social Science Research
1 October 2020 in Zeitschrift Fur Soziologie
Zusammenfassung: Die Differenzierung der Bildungswege nach den kognitiven Fähigkeiten wird damit begründet, dass sie eine für das Lernen förderliche Homogenisierung der schulischen Umgebung ermögliche, die allen beim Erwerb schulischer Kompetenzen zugutekäme. Dagegen wird angeführt, dass die Differenzierung keine sonderlichen Effekte habe, sondern eher nur die Effekte der sozialen Herkunft verstärke. Diese Auffassung gilt als Standardposition. Der Beitrag untersucht auf der Grundlage eines allgemeinen theoretischen Modells die Effekte einer unterschiedlich strikt geregelten Differenzierung auf die Leistungen in der Sekundarstufe. Grundlage sind Daten der „National Educational Panel Study“ (NEPS) für die deutschen Bundesländer mit ihren erheblichen Unterschieden in der Regelung der Differenzierung. Danach gibt es bei einer strikten Differenzierung keine Verstärkung der Effekte der sozialen Herkunft, wohl aber eine Zunahme der Leistungen in der Sekundarstufe, speziell in der Kombination mit einer homogeneren Zusammensetzung der Schulklassen nach den kognitiven Fähigkeiten. Das gilt gerade für die Kinder in den Schulklassen der unteren Bildungswege mit geringerem Leistungsniveau. Dort fallen die Leistungen am geringsten bei kognitiver Homogenität und einer liberalen, am besten bei Homogenität und einer strikten Differenzierung aus. In den Schulklassen mit höherem kognitiven Niveau ist die Streuung dagegen gleichgültig, dort wäre die kognitive Heterogenität eher sogar besser. Die Befunde widersprechen der Standardposition deutlich, womöglich wohl, weil in den bisherigen Analysen die theoretisch erforderlichen Daten nicht zur Verfügung standen oder die nötigen Analysen nicht vorgenommen wurden, insbesondere nicht in den Beiträgen mit den bisher meist verwendeten internationalen Vergleichsstudien.
9 May 2022 in Frontiers in Sociology
Mastering the language of the destination country is key to immigrant and refugee children's educational success. Refugee children typically face the challenge of starting or continuing their educational carrier in a completely new context and in a completely new language. In this study, we examine the role of preschool attendance and formal language instruction in supporting young refugee children to acquire destination language competencies. We pursue three research objectives: First, we aim at identifying relevant conditions associated with German language acquisition in general. Second, we examine the (relative) importance of institutional learning support in preschool and language instruction. Third, we investigate whether the benefits of attending preschool are more pronounced for refugee children who have only limited exposure to the destination language outside of the institutional context, as compared to children who have more exposure to the language outside of preschool. Using data from the ReGES study, we analyze the early processes of destination language acquisition among a large population of refugee children of preschool age in Germany. Our findings indicate that conditions associated with motivation, exposure and efficiency of learning that were found in prior research to determine destination language competencies of children from other immigrant groups apply to refugee children in a similar manner. Additional conditions associated with the specific circumstances that refugees often experience, including possible consequences of insecure residence status, risk of post-traumatic stress disorders, and living in collective accommodation, do not significantly contribute to this outcome in our analysis. Furthermore, we find that there is a positive relationship between children's German language competency levels and both preschool attendance and formal language instruction. The findings indicate that the benefits of attending preschool are largely related to additional language instruction that refugee children receive within this context. Moreover, these benefits are particularly pronounced among refugee children who have only limited exposure to German at home and in their everyday lives. Overall, our findings emphasize the importance of preschool attendance and formal language instruction for refugee children's destination language acquisition.
19 March 2025 in Royal Society Open Science
This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a ‘transparent group’ receiving original study and code or ‘opaque group’ receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p -value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers ‘doing reproduction’ would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in ‘push button’ replications.
17 June 2016 in KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie