1 June 2010 in ACM Transactions on Computing Education
A concept inventory is a standardized assessment tool intended to evaluate a student’s understanding of the core concepts of a topic. In order to create a concept inventory it is necessary to accurately identify these core concepts. A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing subjects: discrete mathematics, programming fundamentals, and logic design. The topic rankings can not only be used to guide the coverage of concept inventories, but can also be used by instructors to identify what topics merit special attention.
2 October 2014 in Computer Science Education
Concept inventories hold tremendous promise for promoting the rigorous evaluation of teaching methods that might remedy common student misconceptions and promote deep learning. The measurements from concept inventories can be trusted only if the concept inventories are evaluated both by expert feedback and statistical scrutiny (psychometric evaluation). Classical Test Theory and Item Response Theory provide two psychometric frameworks for evaluating the quality of assessment tools. We discuss how these theories can be applied to assessment tools generally and then apply them to the Digital Logic Concept Inventory (DLCI). We demonstrate that the DLCI is sufficiently reliable for research purposes when used in its entirety and as a post-course assessment of students’ conceptual understanding of digital logic. The DLCI can also discriminate between students across a wide range of ability levels, providing the most information about weaker students’ ability levels.
3 July 2017 in IEEE Transactions on Education
This paper presents and analyzes results of two Delphi processes that polled cybersecurity experts to rate cybersecurity topics based on importance, difficulty, and timelessness. These ratings can be used to identify core concepts–cross-cutting ideas that connect knowledge in the discipline. The first Delphi process identified core concepts that should be learned in any first course on cybersecurity. The second identified core concepts that any cybersecurity professional should know upon graduating from college. Despite the rapidly growing demand for cybersecurity professionals, it is not clear what defines foundational cybersecurity knowledge. Initial data from the Delphi processes lay a foundation for defining the core concepts of the field and, consequently, provide a common starting point to accelerate the development of rigorous cybersecurity education practices. These results provide a foundation for developing evidence-based educational cybersecurity assessment tools that will identify and measure effective methods for teaching cybersecurity. The Delphi results can also be used to inform the development of curricula, learning exercises, and other educational materials and policies.