The increased reliance on online education and educational technologies more generally has laid bare the need to more deeply consider how researchers, designers, and educators can improve the quality of technology-mediated learning. To address this need, more than two dozen experts from a variety of fields came together to discuss the challenges that educational technology must address in the immediate future. These experts were tasked with identifying barriers to and potential solutions for delivering high-quality and equitable online and remote education. This article examines the themes and topics that emerged from these discussions and proposes a Collaborative Framework for Accelerating Online Education. This framework highlights the need for rapid experimentation within larger design cycles as well as the coordination and cooperation of multiple stakeholders across all phases of research and development. The themes, topics, and framework that emerged from this work serve as a call to action for innovative approaches to developing and studying online education.
1 January 2014 in Proceedings of the American Society for Information Science and Technology
There is growing recognition of the importance of learning as a search outcome and of the need to provide support for it. Yet, before we can consider learning as a part of search, we need to know how to assess it. This panel will focus on methods and measures for assessing learning in the context of search tasks and their outcomes. The panel will be interactive as the audience will be encouraged to engage in contributing their own experiences and ideas related to measures and methods to study learning as a part of search processes. Ideas and experiences with explicit and implicit indicators of learning and with evaluating learning outcomes will be shared during a dialogue between the audience and panelists. Outcomes from the panel discussions will contribute to formulating a research agenda for “search as learning.” The outcomes will be shared with the audience (and the wider ASIST community).
7 April 2016 in Journal of Medical Internet Research
We present an empirical study evaluating the quality of multiple-choice questions (MCQs) generated by Large Language Models (LLMs) from a corpus of video transcripts of course lectures in an online data science degree program. With our database of thousands of generated questions, we conducted both human and automated judging of question quality on a representative sample using a broad set of criteria, including well-established Item Writing Flaw (IWF) categories. We found the number of average IWFs per MCQ ranged from 1.6 (rule-based verification) to 2.18 (LLM-based). Among the most frequently identified MCQ flaws were lack of enough context (17%) or answer choices with at least one implausible distractor (57%). Both human and automated assessment identified implausible distractors as one of the most frequent flaw categories. Results from our human annotation study were generally more positive (51--65% good items) compared to our automated assessment study results, which tended toward greater flaw identification (15--25% good items), depending on evaluation method.