Intelligent tutoring systems have become increasingly common in assisting students but are often aimed at isolated subject-domain tasks without creating a scaffolding system from lower- to higher-level cognitive skills, with low-level skills often neglected. We designed and developed an intelligent tutoring system, CompPrehension, which aims to improve the comprehension level of Bloom’s taxonomy. The system features plug-in-based architecture, easily adding new subject domains and learning strategies. It uses formal models and software reasoners to solve the problems and judge the answers, and generates explanatory feedback about the broken domain rules and follow-up questions to stimulate the students’ thinking. We developed two subject domain models: an Expressions domain for teaching the expression order of evaluation, and a Control Flow Statements domain for code-tracing tasks. The chief novelty of our research is that the developed models are capable of automatic problem classification, determining the knowledge required to solve them and so the pedagogical conditions to use the problem without human participation. More than 100 undergraduate first-year Computer Science students took part in evaluating the system. The results in both subject domains show medium but statistically significant learning gains after using the system for a few days; students with worse previous knowledge gained more. In the Control Flow Statements domain, the number of completed questions correlates positively with the post-test grades and learning gains. The students’ survey showed a slightly positive perception of the system.
When developing automated assessments and intelligent tutoring systems, a lot of routine effort goes into developing the bank of learning problems. Problem generation is the way to automate this process. In this paper, we present a method of generating expression-related problems for teaching introductory programming courses. The problems are generated from open-source software code which allows keeping learning problems similar to the production code the students should learn to analyze and write. Generated problems are automatically classified by their difficulties and the knowledge they need to solve, represented as sets of possible errors. This allows seamless integration with adaptive learning algorithms. The evaluation showed that the generated problems are indistinguishable from human-authored problems and suitable for use in the educational process.
When developing automatic quizzing systems and intelligent tutoring systems, significant effort has to be spent on developing the problem bank. Question and problem generation is a field of study concerning automating this routine work. The two most common methods of problem generation are constrained random generation and template-based generation, but each of them has disadvantages. In this work, we study a possibility of generating code-tracing learning problems from existing open-source code. We generated a set of learning problems and evaluated their distinctness from human-authored problems and readiness for usage in the learning process. Both teachers and students showed the rate of determining machine-generated problems only slightly above random guessing. Teachers strongly agreed with the problems' relevance and agreed with their suitability for the learning process. Automatic labeling to filter desired problems for the assignment includes used concepts, possible errors during solving, and difficulty estimates. The studied type of learning problem required little additional data to add to the code; our further work will concern problem types with more dynamic data to overcome this limitation.