This study investigates the validity and reliability of generative large language models (LLMs), specifically ChatGPT and Google's Bard, in grading student essays in higher education based on an analytical grading rubric. A total of 15 experienced English as a foreign language (EFL) instructors and two LLMs were asked to evaluate three student essays of varying quality. The grading scale comprised five domains: grammar, content, organization, style & expression and mechanics. The results revealed that fine‐tuned ChatGPT model demonstrated a very high level of reliability with an intraclass correlation (ICC) score of 0.972, Default ChatGPT model exhibited an ICC score of 0.947 and Bard showed a substantial level of reliability with an ICC score of 0.919. Additionally, a significant overlap was observed in certain domains when comparing the grades assigned by LLMs and human raters. In conclusion, the findings suggest that while LLMs demonstrated a notable consistency and potential for grading competency, further fine‐tuning and adjustment are needed for a more nuanced understanding of non‐objective essay criteria. The study not only offers insights into the potential use of LLMs in grading student essays but also highlights the need for continued development and research. Practitioner notes: What is already known about this topic Large language models (LLMs), such as OpenAI's ChatGPT and Google's Bard, are known for their ability to generate text that mimics human‐like conversation and writing. LLMs can perform various tasks, including essay grading. Intraclass correlation (ICC) is a statistical measure used to assess the reliability of ratings given by different raters (in this case, EFL instructors and LLMs). What this paper adds The study makes a unique contribution by directly comparing the grading performance of expert EFL instructors with two LLMs—ChatGPT and Bard—using an analytical grading scale. It provides robust empirical evidence showing high reliability of LLMs in grading essays, supported by high ICC scores. It specifically highlights that the overall efficacy of LLMs extends to certain domains of essay grading. Implications for practice and/or policy The findings open up potential new avenues for utilizing LLMs in academic settings, particularly for grading student essays, thereby possibly alleviating workload of educators. The paper's insistence on the need for further fine‐tuning of LLMs underlines the continual interplay between technological advancement and its practical applications. The results lay down a footprint for future research in advancing the use of AI in essay grading.
27 July 2020 in International Journal of Educational Technology in Higher Education
There is a general belief that software must be able to easily do things that humans find difficult. Since finding sources for plagiarism in a text is not an easy task, there is a wide-spread expectation that it must be simple for software to determine if a text is plagiarized or not. Software cannot determine plagiarism, but it can work as a support tool for identifying some text similarity that may constitute plagiarism. But how well do the various systems work? This paper reports on a collaborative test of 15 web-based text-matching systems that can be used when plagiarism is suspected. It was conducted by researchers from seven countries using test material in eight different languages, evaluating the effectiveness of the systems on single-source and multi-source documents. A usability examination was also performed. The sobering results show that although some systems can indeed help identify some plagiarized content, they clearly do not find all plagiarism and at times also identify non-plagiarized material as problematic.
6 March 2023 in International Journal for Educational Integrity
Academic integrity is a vital pedagogical responsibility that educational institutions should explicitly address. One of the best ways to uphold academic integrity is to create a culture of academic integrity throughout the school. This is especially imperative at high schools where students develop their moral identity because students who act dishonestly at high school will likely behave accordingly in post-secondary education and ultimately be dishonest in familial and professional settings. Creating a culture of academic integrity is a challenging, long and multifaceted journey. In this respect, this exploratory case study set out to create a culture of academic integrity at a high school in Türkiye and explore what facilitates and impedes the process. We followed Stephens’ (2016) Multilevel Intervention Model and implemented a School-Wide Education program to guide us through the process. We conducted various activities throughout one academic year, from seminars to competitions. At the end of the term, we conducted individual and focus group interviews with the members of the school community and analyzed the interview data to identify facilitators and barriers of the process. The analysis yielded five facilitators (1) creating buy-in, (2) administrative embracement and support, (3) activities that promote student involvement, (4) external expert and school collaboration as praxis, and (5) policy as the blueprint and five barriers (1) deficiencies in responding academic misconduct, (2) prioritization of academic success over academic integrity, (3) teacher resistance against change, (4) exam-based assessment design, and (5) timing of the activities. Each theme was discussed in detail, and recommendations were made for high schools which set out on a journey of creating academic integrity culture.
30 August 2024 in Sakarya University Journal of Education
Through the last decades, Artificial Intelligence (AI) has revolutionized the field of education and transformed traditional teaching approaches. This study aimed to examine how university students adopt AI tools in their learning processes and the role of digital literacy (DL) in this process through the lens of the Technology Acceptance Model (TAM). In this context, this study measured the impact of DL on university students' acceptance of AI technologies and their intention to use such technologies in the future. The data was collected from university students (N = 154) at a university in Western Türkiye during the fall semester of 2023. Data collection was conducted using two separate online forms; the first form included items adapted from the Digital Literacy Scale developed by Bayrakçı and Narmanlıoğlu (2021) to measure digital literacy levels, while the second form included items adapted from the UTAUT study by Venkatesh et al. (2003). The hypothesis testing results showed that students with higher levels of DL perceived the usefulness and ease of use of AI tools more positively, which positively affected their intention to adopt AI-based tools. The study also found that perceived usefulness and ease of use were important in shaping students' attitudes and behavioural intentions towards AI. When students perceive AI as a valuable tool for learning and find it easy to interact with, they are more willing to use it. This study suggests that DL plays a significant role in the acceptance of AI-based tools among university students, and accordingly, the TAM is a practical and accurate model to explore students’ potential engagement with AI in the learning process.
As artificial intelligence (AI) increasingly influences education, understanding learners' experiences, engagement and literacy of these tools is critical. This study explores AI literacy among Turkish EFL (English as a Foreign Language) students regarding their familiarity, knowledge, and ethical perceptions of AI technologies in academic writing. Using a descriptive exploratory approach, the study surveyed 427 students from two Turkish universities. Findings reveal a moderate level of AI familiarity and usage among participants, with a significant reliance on AI tools for translation and grammar proofreading. Despite recognizing AI's potential to enhance academic writing, students exhibited limited technical proficiency and understanding of AI's underlying mechanisms, highlighting a need for targeted and structured AI education for EFL writing. The findings contribute to the ongoing discourse on AI integration in EFL education, offering insights for policymakers, educators, and researchers to better prepare students for an AI-driven academic environment.
30 April 2020 in World Journal on Educational Technology: Current Issues
Writing is one of the most difficult and neglected skills for EFL learners. Gamification, the use of game elements to increase user experience and interest in non-game content, can be used to deal with writing anxiety which is considered as a hindering affective factor in successful language learning. Within this scope, this study aims to investigate the effect of the online gamification tool "˜Edmodo" on EFL learners' writing anxiety levels. The experimental design, one of the quantitative methods, was preferred. Data were collected through a background questionnaire and the Second Language Writing Anxiety Scale, involving 22 items scored on a five-point Likert response scale. Data were analysed with the Statistical Package for Social Sciences software. The results showed that the students who completed the activities and who were graded through Edmodo had significantly lower anxiety levels than the students using the traditional pen and paper. Keywords: Gamification, foreign language, writing, anxiety.