Techniques emerging from the considerable research on cognitive aspects of survey methodology include various forms of probing and cognitive interviewing. These techniques are used to examine whether respondents' interpretations of self-report items are consistent with researchers' assumptions and intended meanings given the constructs the items are designed to measure. However, although informal procedures are common, such developments have not been systematically applied in educational research. We describe how information derived from the systematic application of cognitive pretesting can contribute to determining the validity—designated cognitive validity—of self-report items. Examples are presented from prominent motivation-related instruments that assess real-world instructional practices, mastery classroom goal structure, and student self-efficacy. The implications and pragmatics of adopting this approach are discussed.

This study investigated the prevalence of transformative experiences, antecedents of transformative experience, and the relation between transformative experience and deep‐level learning (conceptual change and transfer) for high school biology students (N = 166). Results suggested that the high school students in our sample typically engaged in low levels of transformative experience with respect to biology, but those students who strongly identified with science and who endorsed a mastery goal orientation were more likely to report engagement in higher levels of transformative experience. Furthermore, a higher level of engagement in transformative experience was positively associated with (a) conceptual change in understanding the concept of natural selection, but not inheritance, at the post‐ and follow‐up assessments and (b) transfer at the follow‐up assessment. © 2009 Wiley Periodicals, Inc. Sci Ed 94:1–28, 2010

The Teaching for Transformative Experiences in Science (TTES) model is designed to foster transformative experiences (e.g., experiences with science content involving the application of that content in ways that expand perception and value in everyday experience). This study presents a case study of a high school biology teacher learning to implement the TTES model in conjunction with a conceptual change model and evaluates the impact of these models on students’ learning and engagement. In collaboration with the authors, the teacher created three instructional conditions: (a) control (2 classes, 40 students), (b) cognitive conflict (CC, 2 classes, 42 students), and (c) combined TTES and cognitive conflict (TE, 2 classes, 44 students). Descriptive data illustrate how the teacher implemented the models, particular challenges that arose, how the teacher's prior beliefs and goals shaped the implementation, and how experience with the models altered the teacher's beliefs and engagement with the content. Comparative analyses across conditions illustrate that students in the TE condition displayed more transformative engagement in class but did not report a greater level of independent transformative experiences. Regular-level students in the intervention conditions (CC, TE) displayed greater change in basic knowledge and regular-level students in the TE condition displayed more enduring transfer. Honors-level students in the intervention conditions (CC, TE) displayed greater conceptual change, but not greater transfer.

Onwuegbuzie and colleagues proposed the Instrument Development and Construct Validation (IDCV) process as a mixed methods framework for creating and validating measures. Examples applying IDCV are lacking. We provide an illustrative case integrating the Rasch model and cognitive interviews applied to the development of the Transformative Experience Questionnaire (TEQ). TEQ measures student engagement with the content they are learning. IDCV process yielded a high degree of utility for informing the psychometric properties of the TEQ. The quantitative data yielded supportive content, construct, and concurrent-related validity evidence using a sample of middle (n = 115) and high school students (n = 81). The qualitative data revealed item rewording was needed. Moderate to strong test–retest reliability was found for both forms.