Practitioners and evaluators face several constraints in conducting rigorous evaluations to determine program effect. Researchers have offered the retrospective pretest/posttest design as a remedy to curb response-shift bias and better estimate program effects. This article presents an example of how After School Matters (ASM) tested the use of retrospective pretest/posttest design for evaluating out-of-school time (OST) programs for high school youth participants. Differences between traditional pretest and retrospective pretest scores were statistically significant, but effect sizes were negligible, indicating that both pretests yielded similar results. Interviews with youth led to 3 key findings that have implications for ASM using retrospective pretests with youth: response-shift bias was more prominent in youth interviews than in quantitative findings, youth recommended reordering the questions so that the retrospective pretest appears first to increase comprehension, and acquiescence bias emerged in the interviews. This study demonstrates that the retrospective pretest/posttest design can be an alternative to the traditional pretest/posttest design for OST at ASM. These findings are important for ASM and other youth-serving organizations, which often have limited capacity to survey youth multiple times within 1 program session.
The evaluation theory tree typology reflects the following three components of evaluation practice: (a) methods, (b) use, and (c) valuing. The purpose of this study was to explore how evaluation practice is conceived as reflected in articles published in the American Journal of Evaluation (AJE) and Evaluation, a journal supported by the European Evaluation Society. A key finding from this international comparison suggests that evaluation practice as reflected in AJE and Evaluation both emphasize methods, in comparison to use and valuing. This article concludes using Peter Dahler-Larsen’s discussion on evaluation societies, among other sources, to examine the audit society, which might account for the trends in our findings. EvalPartners, a global community of evaluators, has declared 2015 as the international year of evaluation. These findings regarding cross-continental trends in evaluation are relevant for engaging in a global dialogue on evaluation practice.
10 November 2009 in American Journal of Evaluation