10 July 2019 in IEEE Transactions on Education
Contribution: An automated methodology that provides visualizations of students' free text comments from course satisfaction surveys. Focusing on sentiment, these visualizations reveal learning and teaching aspects of the course that either may require improvement or are performing well. They provide educators with a simple, systematic way to monitor their courses and make pedagogically sound decisions on teaching strategies. Background: Student course satisfaction surveys often solicit free text comments. This feedback can provide invaluable insights for educators, but because these comments often contain a large amount of data, they cannot easily be acted upon. Existing visualization methods are not suitable for this application, and needed additional capabilities. Research Questions: How can large quantities of student satisfaction data be summarized and visualized? How can these visualizations be used to learn meaningful information about courses? What are the recurring themes across semesters? Methodology: Several methods based on machine learning and text analysis techniques were used to visualize student satisfaction comments. The latent Dirichlet allocation (LDA) statistical method was used to identify aspects of student opinion of a course. The sentiment of the student comments were also identified. This information was then presented visually for educators in a case study that gives examples of these visualizations. Findings: The visualization methods explored provide educators with an overview of aspects and their associated sentiment. The summary visualizations allow easy comparison to be made between courses, or between teaching periods in the same course.
6 December 2021 in Australasian Journal of Educational Technology
In this study, we applied natural language processing (NLP) techniques, within an educational environment, to evaluate their usefulness for automated assessment of students’ conceptual understanding from their short answer responses. Assessing understanding provides insight into and feedback on students’ conceptual understanding, which is often overlooked in automated grading. Students and educators benefit from automated formative assessment, especially in online education and large cohorts, by providing insights into conceptual understanding as and when required. We selected the ELECTRA-small, RoBERTa-base, XLNet-base and ALBERT-base-v2 NLP machine learning models to determine the free-text validity of students’ justification and the level of confidence in their responses. These two pieces of information provide key insights into students’ conceptual understanding and the nature of their understanding. We developed a free-text validity ensemble using high performance NLP models to assess the validity of students’ justification with accuracies ranging from 91.46% to 98.66%. In addition, we proposed a general, non-question-specific confidence-in-response model that can categorise a response as high or low confidence with accuracies ranging from 93.07% to 99.46%. With the strong performance of these models being applicable to small data sets, there is a great opportunity for educators to implement these techniques within their own classes. Implications for practice or policy: Students’ conceptual understanding can be accurately and automatically extracted from their short answer responses using NLP to assess the level and nature of their understanding. Educators and students can receive feedback on conceptual understanding as and when required through the automated assessment of conceptual understanding, without the overhead of traditional formative assessment. Educators can implement accurate automated assessment of conceptual understanding models with fewer than 100 student responses for their short response questions.
13 July 2022 in Assessment & Evaluation in Higher Education
Student evaluation of teaching (SET) surveys are the most widely used tool for collecting higher education student feedback to inform academic quality improvement, promotion and recruitment processes. Malicious and abusive student comments in SET surveys have the potential to harm the wellbeing and career prospects of academics. Despite much literature highlighting abusive feedback in SET surveys, little research attention has been given to methods for screening student comments to identify and remove those that may cause harm to academics. This project applied innovative machine learning techniques, along with a dictionary of keywords to screen more than 100,000 student comments made via a university SET during 2021 Higher Education Standards Framework (Threshold Standards). 2021. (Cth) (Austrl.) [Google Scholar] . The study concluded that these methods, when used in conjunction with a final stage of human checking, are an effective and practicable means of screening student comments. Higher education institutions have an obligation to balance the rights of students to provide feedback on their learning experience with a duty to protect academics from harm by pre-screening student comments before releasing SET results to academics.
13 July 2020 in Computer Applications in Engineering Education
6 April 2023 in Journal of Higher Education Policy and Management
Teaching evaluation is deeply entrenched in institutional quality assurance and is a feature of a range of policies including academic recruitment, promotion, and performance management. Universities must ensure that evaluation practices meet regulatory requirements, while balancing student voice and wellbeing. There is extensive literature examining the validity, reliability, and bias of student surveys, but limited focus on institutional evaluation strategies and what constitutes best practice. This study analyses and reports on the teaching evaluation strategies and practices of Australian and New Zealand universities. All 29 participating institutions use standardised and centrally deployed surveys to evaluate teaching and participated in external student experience surveys to benchmark nationally. Comparisons are provided between the strategic intent of evaluation; survey practices, technology, analysis, and reporting; and the use of other nonsurvey methods. The study informs higher education institutions’ evaluation strategy, policies, and practice and provides an agenda for reframing approaches to evaluation of teaching.
11 August 2020 in Assessment & Evaluation in Higher Education
This study examines the utility of a new method of analysing and reporting qualitative student survey comments. Visualisations of the sentiment of and key themes from qualitative student survey comments were developed for 34 subjects from two undergraduate health sciences courses undergoing transformation. The course and subject visualisation reports were provided to academics who were members of the two course transformation teams. Utilising focus groups and semi-structured interview methodology, we examined academic perspectives on the potential usefulness of this new method of analysing and reporting qualitative comments from student evaluations of teaching (SET). Results indicate that visualisations are considered useful for focussing educators’ attention on themes in qualitative comments rather than on individual negative comments. However, educators did not support use of the visualisations at this stage, citing concerns about the accuracy of sentiment analysis and the validity of SET surveys. There were many opportunities identified through the focus groups for future improvements to a visualisation technique for reporting student survey results.