Students taking computing courses develop skills by applying programming to various problems. In the past decade, courses have started to move from manually graded assessments to those which can be automated. A typical motivation for the use of automated assessment is to enable scaling, which is of particular importance for courses taught with high enrollment. However, automated assessment provides additional advantages such as reproducibility and rapid feedback. We have developed an automated assessment platform that performs a combination of static and dynamic analysis to evaluate student work. Our focus has been not on scaling but rather on serving student educational outcomes, both at the student level (e.g., providing feedback according to teaching best practices) and program level (e.g., ensuring that students across semesters meet the same standard). In this paper, we report on the design, development, and introduction of an automated assessment tool to improve instruction. The use of this tool has been aligned with a course on data structures & algorithms taught at the sophomore level. We discuss the development of the tool, the techniques that it uses, and evaluate its impact on grade accuracy.
Problem solving is a necessary skill for computing students to succeed in industry. Students commonly find it difficult to apply their technical skills in solving real-world problems, which can be addressed by the inclusion and scaffolding of open-ended and ill-defined problems in formative computing courses. An Analysis, Design, and Justification (ADJ) based framework was recently developed to enable this goal. The ADJ framework consists of learning outcomes, a type of problem, and a three-step problem-solving process. ADJ supports instruction in environments with constraints (such as large classes, and heterogeneous student ability) that are difficult to support with other approaches (such as Problem-Based Learning). In this paper, we report the experience and impact of leveraging the ADJ framework to develop real-world problem-solving skills in the context of a formative computing course. The ADJ framework was enacted in a class on operating-systems taught at the junior level. In the class, students were taught the concepts of the ADJ framework using a series of discussion activities, and then asked to apply it to generate solutions to three problem sets. The problem sets contained problems on topic areas covered by other assessments in the course but used open-ended and ill-defined problems. Assessments showed that students had different performance on existing (well-defined) problems as opposed to the real-world style problems of ADJ. The ADJ assessments provided a step towards better evaluation of problem-solving ability.
This innovative practice full paper describes the challenges of teaching operating systems programming and how automated assessment can be introduced to address them. Operating systems courses help students understand the role of run-time environments in executing software and often introduce topics like thread management and terminal application development. To teach the fundamentals of operating systems, these courses are usually taught in a language like C that provides low-level access to resources. Based on the challenges of using a low-level language, we have developed an automated assessment framework using code instrumentation and static analysis techniques that extend the commonly used unit testing style methodology wherein various test cases are created for student functions. Our techniques allow the assessment of additional requirements such as effective memory management. Although there is substantial work on applying automated assessment in introductory courses (often taught in Java), it has yet to gain prominence in advanced courses. It has been shown to improve student performance, further motivating its use in more advanced courses. As a step towards leveraging automated assessment, we discuss the design of an automated assessment tool for a programming assignment that focuses on practicing memory management. We perform an initial evaluation of its use in the operating systems course at our university to demonstrate the assessment accuracy. Our work contributes a discussion of the challenges involved in teaching operating systems programming, how those challenges can be addressed with automated assessment, and a framework for integrating automated assessment into low-level course assignments.
This full research paper investigates the evolution of code quality in student submissions within a second-year Data Structures and Algorithms (DS&A) course taught in Java. Poorly written code in the industry often leads to significant technical debt, making future modifications and scalability challenging. To address this issue at the educational level, our study aims to understand how students' coding practices naturally progress throughout our course and identify patterns that can inform pedagogical strategies for enhancing programming skills. A set of code quality metrics, forms the basis of this analysis. We analyze student submissions across multiple course modules and track select code quality metrics tailored to their second-year level. These results are analyzed at various granularities - modulewise and metric-wise trends - to capture the nuances of students' coding evolution. Our findings show that, without additional guidance, code quality improves over time, with reductions in readability and maintainability errors as the course progresses. Formatting errors, such as whitespace violations, remain high, comprising nearly half of all detected issues. The analysis reveals correlations between automated feedback on the assignment's technical requirements and subsequent improvements in code quality metrics, highlighting the natural evolution of students' practices as they adapt to increasingly complex programming tasks. The findings contribute to an understanding of students' learning behaviors in a DS&A course and provide insights for engineering educators to improve instructional approaches while establishing a framework for integrating code quality into computer science and software engineering education that can be adapted by institutions teaching fundamental programming concepts.