2 January 2020 in Studies in Science Education
Machine learning (ML) is an emergent computerised technology that relies on algorithms built by ‘learning’ from training data rather than ‘instruction’, which holds great potential to revolutionise science assessment. This study systematically reviewed 49 articles regarding ML-based science assessment through a triangle framework with technical, validity, and pedagogical features on three vertices. We found that a majority of the studies focused on the validity vertex, as compared to the other two vertices. The existing studies primarily involve text recognition, classification, and scoring with an emphasis on constructing scientific explanations, with a vast range of human-machine agreement measures. To achieve the agreement measures, most of the studies employed a cross-validation method, rather than self- or split-validation. ML allows complex assessments to be used by teachers without the burden of human scoring, saving both time and cost. Most studies used supervised ML, which relies on extraction of attributes from student work that was first coded by humans to achieve automaticity, rather than semi- or unsupervised ML. We found that 24 studies were explicitly embedded in science learning activities, such as scientific inquiry and argumentation, to provide feedback or learning guidance. This study identifies existing research gaps and suggests that all three vertices of the ML triangle should be addressed in future assessment studies, with an emphasis on the pedagogy and technology features.
1 July 2004 in Biochimica et Biophysica Acta (BBA) - General Subjects
6 October 2020 in Journal of Research in Science Teaching
1 September 2012 in CBE—Life Sciences Education
Students’ writing can provide better insight into their thinking than can multiple-choice questions. However, resource constraints often prevent faculty from using writing assessments in large undergraduate science courses. We investigated the use of computer software to analyze student writing and to uncover student ideas about chemistry in an introductory biology course. Students were asked to predict acid–base behavior of biological functional groups and to explain their answers. Student explanations were rated by two independent raters. Responses were also analyzed using SPSS Text Analysis for Surveys and a custom library of science-related terms and lexical categories relevant to the assessment item. These analyses revealed conceptual connections made by students, student difficulties explaining these topics, and the heterogeneity of student ideas. We validated the lexical analysis by correlating student interviews with the lexical analysis. We used discriminant analysis to create classification functions that identified seven key lexical categories that predict expert scoring (interrater reliability with experts = 0.899). This study suggests that computerized lexical analysis may be useful for automatically categorizing large numbers of student open-ended responses. Lexical analysis provides instructors unique insights into student thinking and a whole-class perspective that are difficult to obtain from multiple-choice questions or reading individual responses.
26 September 2020 in Journal of Science Education and Technology
We systematically compared two coding approaches to generate training datasets for machine learning (ML): (i) a holistic approach based on learning progression levels and (ii) a dichotomous, analytic approach of multiple concepts in student reasoning, deconstructed from holistic rubrics. We evaluated four constructed response assessment items for undergraduate physiology, each targeting five levels of a developing flux learning progression in an ion context. Human-coded datasets were used to train two ML models: (i) an 8-classification algorithm ensemble implemented in the Constructed Response Classifier (CRC), and (ii) a single classification algorithm implemented in LightSide Researcher’s Workbench. Human coding agreement on approximately 700 student responses per item was high for both approaches with Cohen’s kappas ranging from 0.75 to 0.87 on holistic scoring and from 0.78 to 0.89 on analytic composite scoring. ML model performance varied across items and rubric type. For two items, training sets from both coding approaches produced similarly accurate ML models, with differences in Cohen’s kappa between machine and human scores of 0.002 and 0.041. For the other items, ML models trained with analytic coded responses and used for a composite score, achieved better performance as compared to using holistic scores for training, with increases in Cohen’s kappa of 0.043 and 0.117. These items used a more complex scenario involving movement of two ions. It may be that analytic coding is beneficial to unpacking this additional complexity.
2 July 2022 in Research in Science Education