Peer assessment typically requires students to judge peers' work against assessment criteria. We tested an alternative approach in which students judged pairs of scripts against one another in the absence of assessment criteria. First year mathematics undergraduates (N = 194) sat a written test on conceptual understanding of multivariable calculus, then assessed their peers' responses using pairwise comparative judgement. Inter-rater reliability was investigated by randomly assigning the students to two groups and correlating the two groups' assessments. Validity was investigated by correlating the peers' assessments with (i) expert assessments, (ii) novice assessments, and (iii) marks from other module tests. We found high validity and inter-rater reliability, suggesting that the students performed well as peer assessors. We interpret the results in the light of survey and interview feedback, and discuss directions for further research into the benefits and drawbacks of peer assessment without assessment criteria.
18 January 2014 in International Journal of Science and Mathematics Education
There is an increasing demand from employers and universities for school leavers to be able to apply their mathematical knowledge to problem solving in varied and unfamiliar contexts. These aspects are however neglected in most examinations of mathematics and, consequentially, in classroom teaching. One barrier to the inclusion of mathematical problem solving in assessment is that the skills involved are difficult to define and assess objectively. We present two studies that test a method called comparative judgement (CJ) that might be well suited to assessing mathematical problem solving. CJ is an alternative to traditional scoring that is based on collective expert judgements of students’ work rather than item-by-item scoring schemes. In study 1, we used CJ to assess traditional mathematics tests and found it performed validly and reliably. In study 2, we used CJ to assess mathematical problem-solving tasks and again found it performed validly and reliably. We discuss the implications of the results for further research and the implications of CJ for the design of mathematical problem-solving tasks.
School mathematics examination papers are typically dominated by short, structured items that fail to assess sustained reasoning or problem solving. A contributory factor to this situation is the need for student work to be marked reliably by a large number of markers of varied experience and competence. We report a study that tested an alternative approach to assessment, called comparative judgement, which may represent a superior method for assessing open-ended questions that encourage a range of unpredictable responses. An innovative problem solving examination paper was specially designed by examiners, evaluated by mathematics teachers, and administered to 750 secondary school students of varied mathematical achievement. The students’ work was then assessed by mathematics education experts using comparative judgement as well as a specially designed, resource-intensive marking procedure. We report two main findings from the research. First, the examination paper writers, when freed from the traditional constraint of producing a mark scheme, designed questions that were less structured and more problem-based than is typical in current school mathematics examination papers. Second, the comparative judgement approach to assessing the student work proved successful by our measures of inter-rater reliability and validity. These findings open new avenues for how school mathematics, and indeed other areas of the curriculum, might be assessed in the future.
1 September 2012 in Journal of Experimental Child Psychology
A sophisticated and flexible understanding of the equals sign (=) is important for arithmetic competence and for learning further mathematics, particularly algebra. Research has identified two common conceptions held by children: the equals sign as an operator and the equals sign as signaling the same value on both sides of the equation. We argue here that, in addition to these two conceptions, the notion of substitution is also an important part of a sophisticated understanding of mathematical equivalence. We provide evidence from a cross-cultural study in which English and Chinese children were asked to rate the "cleverness" of operational, sameness, and substitutive definitions of the equals sign. A principal components analysis revealed that the substitutive items were distinct from the sameness items. Furthermore, Chinese children rated the substitutive items as 'sort of clever' or 'very clever', whereas English children rated them as not so clever, suggesting that the notion of substitution develops differently across the two countries. Implications for developmental models of children's understanding of equivalence are discussed.
1 December 2015 in Studies in Educational Evaluation
This paper was accepted for publication in the journal Studies in Educational Evaluation and the definitive published version is available at http://dx.doi.org/10.1016/j.stueduc.2015.09.004.Peer assessment exercises yield varied reliability and validity. To maximise reliability and validity, the literature recommends adopting various design principles including the use of explicit assessment criteria. Counter to this literature, we report a peer assessment exercise in which criteria were deliberately avoided yet acceptable reliability and validity were achieved. Based on this finding, we make two arguments. First, the comparative judgement approach adopted can be applied successfully in different contexts, including higher education and secondary school. Second, the success was due to this approach; an alternative technique based on absolute judgement yielded poor reliability and validity. We conclude that sound outcomes are achievable without assessment criteria, but success depends on how the peer assessment activity is designed
11 August 2023 in International Journal of Research & Method in Education