The tradition of qualitative research drives much of chemistry education research activity. When performing qualitative studies, researchers must demonstrate the trustworthiness of their analysis so researchers and practitioners consuming their work can understand if and how the presented research claims and conclusions might be transferable to their unique educational settings. There are a number of steps researchers can take to demonstrate the trustworthiness of their work, one of which is demonstrating and reporting evidence of reliability. The purpose of this methodological review is to investigate the methods researchers use to establish and report reliability for chemistry education research articles including a qualitative research component. Drawing from the literature on qualitative research methodology and content analysis, we describe the approaches for establishing the reliability of qualitative data analysis using various measures of inter-rater reliability and processes including negotiated agreement. We used this background literature to guide our review of research articles containing a qualitative component and published in Chemistry Education Research and Practice and the Journal of Chemical Education from the years 2010 through 2019 for whether they report evidence of reliability. We followed this by a more in-depth analysis of how articles from the years 2017 through 2019 discuss reliability. Our analysis indicates that, overall, researchers are presenting evidence of reliability in chemistry education research (CER) articles by reporting reliability measures, describing a process of negotiated agreement, or mentioning reliability and the steps taken to demonstrate it. However, there is a reliance on reporting only percent agreement, which is not considered an acceptable measure of reliability when used on its own. In addition, the descriptions of how reliability was established were not always clear, which may make it difficult for readers to evaluate the veracity of research findings. Our findings indicate that, as a field, CER researchers should be more cognizant of the appropriateness of how we establish reliability for qualitative analysis and should more clearly present the processes by which reliability was established in CER manuscripts.
Teaching organic chemistry requires supporting learning strategies that meaningfully engage students with the challenging concepts and advanced problem-solving skills needed to be successful. Such meaningful learning experiences should encourage students to actively choose to incorporate new concepts into their existing knowledge frameworks by appealing to the cognitive, affective, and psychomotor domains of learning. This study provides a qualitative analysis of students’ meaningful learning experiences after completing three Writing-to-Learn (WTL) assignments in an organic chemistry laboratory course. The assignments were designed to appeal to the three domains necessary for a meaningful learning experience, and this research seeks to understand if and how the WTL assignments promoted students’ meaningful learning. The primary data collected were the students’ responses to open-ended feedback surveys conducted after each assignment. These responses were qualitatively analyzed to identify themes across students’ experiences about their meaningful learning. The feedback survey analysis was triangulated with interviews conducted after each assignment. The results identify how the assignments connected to students’ existing knowledge from other courses and indicate that assignment components such as authentic contexts, clear expectations, and peer review supported students’ meaningful learning experiences. These results inform how assignment design can influence students’ learning experiences and suggest implications for how to support students’ meaningful learning of organic chemistry through writing.
10 July 2020 in Chemistry Education Research and Practice
Learning to reason through organic reaction mechanisms is challenging for students because of the volume of reactions covered in introductory organic chemistry and the complexity of conceptual knowledge and reasoning skills required to develop meaningful understanding. However, understanding reaction mechanisms is valuable for students because they are useful for predicting and explaining reaction outcomes. To identify the features students find pertinent when explaining reaction mechanisms, we have collected students’ written descriptions of an acid-catalysed amide hydrolysis reaction. Students’ writing was produced during the implementation of Writing-to-Learn assignments in a second semester organic chemistry laboratory course. We analysed students’ written responses using an analytical framework for recognizing students’ mechanistic reasoning, originally developed with attention to the philosophy of science literature. The analysis sought to identify the presence of specific features necessary for mechanistic reasoning belonging to four broad categories: (1) describing an overview of the reaction, (2) detailing the setup conditions required for the mechanism to occur, (3) describing the changes that take place over the course of the mechanism, and (4) identifying the properties of reacting species. This work provides a qualitative description of the variety of ways in which students included these features necessary for mechanistic reasoning in their writing. We additionally analysed instances of co-occurrence for these features in students’ writing to make inferences about students’ mechanistic reasoning, defined here as the use of chemical properties to justify how electrons, atoms, and molecules are reorganized over the course of a reaction. Feature co-occurrences were quantified using the lift metric to measure the degree of their mutual dependence. The quantitative lift results provide empirical support for the hierarchical nature of students’ mechanistic descriptions and indicate the variation in students’ descriptions of mechanistic change in conjunction with appeals to chemistry concepts. This research applies a framework for identifying the features present in students’ written mechanistic descriptions, and illustrates the use of an association metric to make inferences about students’ mechanistic reasoning. The findings reveal the capacity of implementing and analysing writing to make inferences about students’ mechanistic reasoning.