Across all sciences, the quality of measurements is important. Survey measurements are only appropriate for use when researchers have validity evidence within their particular context. Yet, this step is frequently skipped or is not reported in educational research. This article briefly reviews the aspects of validity that researchers should consider when using surveys. It then focuses on factor analysis, a statistical method that can be used to collect an important type of validity evidence. Factor analysis helps researchers explore or confirm the relationships between survey items and identify the total number of dimensions represented on the survey. The essential steps to conduct and interpret a factor analysis are described. This use of factor analysis is illustrated throughout by a validation of Diekman and colleagues’ goal endorsement instrument for use with first-year undergraduate science, technology, engineering, and mathematics students. We provide example data, annotated code, and output for analyses in R, an open-source programming language and software environment for statistical computing. For education researchers using surveys, understanding the theoretical and statistical underpinnings of survey validity is fundamental for implementing rigorous education research.
The article presents an instrument measuring university students’ sense of belonging to and involvement in their "home" department (biology), as well as initial validity evidence supporting the proposed use of the instrument.
19 May 2020 in International Journal of STEM Education
Background: Boosting students’ disciplinary interest has long been considered an important mechanism to increase student success and retention in STEM education. Yet, interest is a complex construct and can mean different things to different people, and many of the existing interest questionnaires do not identify a specific theoretical framework underlying their items. To demonstrate that curricular interventions targeting students’ interest are effective, educators need a theoretically based instrument to measure interest. The aim of this study was to develop an instrument measuring undergraduate students’ interest in the discipline of biology and collect initial validity evidence supporting the proposed use. The instrument structure is based on Hidi and Renninger’s (Educational Psychologist 41:111–127, 2006) conceptualization of individual interest, and the intended use is to evaluate changes in the biology interests of the US undergraduate students pursuing STEM degrees. To provide evidence of validity, the instrument was completed by 446 biology majors and 489 non-biology majors at two R1 universities. Exploratory and confirmatory factor analyses were applied to evaluate the internal structure of the instrument.Results: The final three-factor instrument supported by these analyses includes 6 items representing positive feelings towards biology, 5 items representing personal value of biology, and 8 items representing reengagement in biology-related activities. Measurement invariance across biology and non-biology majors was established and subsequent comparisons of these populations demonstrated that biology majors report significantly higher positive feelings, personal value, and reengagement in biology-related activities compared to non-biology majors.Conclusions: The study findings support the use of the instrument to gain a broad understanding of students’ individual interest in biology. With minor adaptions, the instrument could also be evaluated for use in other STEM disciplines and for use by other populations.
10 October 2014 in Journal of Psychoeducational Assessment
The aim of this study was to evaluate the psychometric properties of an expectancy-value-based questionnaire measuring five aspects of test-taking motivation (effort, expectancies, importance, interest, and test anxiety). The questionnaire was distributed to a sample of Swedish Grade 9 students taking a low-stakes (n = 1,047) or a high-stakes (n = 536) test. Structural equation modeling largely supported theoretical assumptions and psychometric properties were acceptable. Expectancies and importance were significant predictors of effort, and effort significantly predicted performance in both test situations. Furthermore, measurement invariance across test situations was investigated. Partial scalar invariance was obtained and latent mean comparisons showed significantly higher levels of reported test-taking motivation in the high-stakes context. The study findings suggest that the present instrument can be used in low- as well as high-stakes assessment contexts not only to get an extended knowledge of test-takers’ perceptions of tests but also to control for lack of motivation when stakes are low for test-takers.
Understanding how students develop biology interests and the roles interest plays in biology contexts could help instructors and researchers to increase science, technology, engineering, and mathematics students’ motivation and persistence. However, it is currently unclear how interest has been defined or measured in the biology education research literature. We analyzed this body of literature to determine how interest has been defined and used by the biology education research community. Specifically, we determined the extent to which previously published work drew on theories that conceptualize interest. Further, we identified studies that measured student interest in biology and characterized the types of measures used. Our findings indicate that biology education researchers typically describe interest as a relationship involving positive feelings between an individual and a physical object, activity, or topic of focus. We also found that interest is often not defined, theories involving interest are not often consulted, and the most common measures of interest only assess a single aspect of the construct. On the basis of these results, we make suggestions for future research seeking to examine biology students’ interest. We hope that this analysis can serve as tool for biology educators to improve their own investigations of students’ interest and measure outcomes of interest-generating educational activities.
18 July 2018 in Journal of College Student Retention: Research, Theory & Practice
The aim of this case study was to explore the sense of belonging to and involvement in a biology department for students participating in a biology program designed to create a strong sense of belonging with high levels of involvement and higher student retention rates. The study is based on 10 semistructured interviews with biology majors. Analysis showed that the students have created a strong sense of community among peers and good relationships with specific faculty and staff. However, the positive social environment did not fully translate into sense of belonging to and involvement in the biology department. Analysis also implied a strong focus on careers related to medical school among students and that students not sharing this strong focus might feel disconnected from the group. Taken together, the interviews identify several different aspects for departments to consider as a way to enhance students’ sense of belonging and involvement.