6 December 2021 in Australasian Journal of Educational Technology
In this study, we applied natural language processing (NLP) techniques, within an educational environment, to evaluate their usefulness for automated assessment of students’ conceptual understanding from their short answer responses. Assessing understanding provides insight into and feedback on students’ conceptual understanding, which is often overlooked in automated grading. Students and educators benefit from automated formative assessment, especially in online education and large cohorts, by providing insights into conceptual understanding as and when required. We selected the ELECTRA-small, RoBERTa-base, XLNet-base and ALBERT-base-v2 NLP machine learning models to determine the free-text validity of students’ justification and the level of confidence in their responses. These two pieces of information provide key insights into students’ conceptual understanding and the nature of their understanding. We developed a free-text validity ensemble using high performance NLP models to assess the validity of students’ justification with accuracies ranging from 91.46% to 98.66%. In addition, we proposed a general, non-question-specific confidence-in-response model that can categorise a response as high or low confidence with accuracies ranging from 93.07% to 99.46%. With the strong performance of these models being applicable to small data sets, there is a great opportunity for educators to implement these techniques within their own classes. Implications for practice or policy: Students’ conceptual understanding can be accurately and automatically extracted from their short answer responses using NLP to assess the level and nature of their understanding. Educators and students can receive feedback on conceptual understanding as and when required through the automated assessment of conceptual understanding, without the overhead of traditional formative assessment. Educators can implement accurate automated assessment of conceptual understanding models with fewer than 100 student responses for their short response questions.
6 January 2023 in IEEE Transactions on Learning Technologies
Academic misconduct stemming from file-sharing websites is an increasingly prevalent challenge in tertiary education, including information technology and engineering disciplines. Current plagiarism detection methods (e.g. text matching) are largely ineffective for combatting misconduct in programming and mathematics based assessments. For these reasons, the development of an effective, automated monitoring tool would provide significant benefit in the struggle against misconduct. To address this challenge, this paper reports an innovative software tool named AssignmentWatch which supports academic integrity by actively monitoring for the upload of assessment content online, with a focus on file-sharing websites. This monitoring and alert notification system reduces the time burden on educators by automating the detection process. The design of AssignmentWatch focuses on early detection which enables educators to take early action. This could include preventative education, or removal of content before it is utilized by students. Through this, AssignmentWatch can reduce incidents of misconduct. The software tool is open source and made freely available to the community. AssignmentWatch was validated under controlled conditions, followed by field testing in 30 assignments, across 16 subjects and 5 higher education institutions. AssignmentWatch was found to effectively detect uploads on a variety of websites, generally within 24 hours. In field testing, AssignmentWatch received positive feedback from educators with 8 out of 10 educators stating AssignmentWatch aided academic integrity and 7 out of 10 users stating AssignmentWatch helped identify assignment content online.