Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, 6200 MD Maastricht, The Netherlands
Dr. Marjan J. B. Govaerts
Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, 6200 MD Maastricht, The Netherlands
Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees’ performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing––using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed.
17 May 2012 in Advances in Health Sciences Education
Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters’ implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters’ information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.
Context: Medical students are expected to self‐regulate their learning within complex and unpredictable clinical learning environments. Research increasingly focuses on the effects of social interactions on the development of self‐regulation in workplace settings, a notion embodied within the concept of co‐regulated learning (CRL). Creating workplace learning environments that effectively foster lifelong self‐regulated learning (SRL) requires a deeper understanding of the relationship between CRL and SRL. The aim of this study was therefore to explore medical students’ perceptions of CRL in clinical clerkships and its perceived impact on the development of their SRL. Methods: We conducted semi‐structured interviews with 11 purposively sampled medical students enrolled in clinical clerkships at one undergraduate competency‐based medical school. Data collection and analysis were conducted iteratively, informed by principles of constructivist grounded theory. Data analysis followed stages of open, axial and selective coding, which enabled us to conceptualise how co‐regulation influences the development of students’ self‐regulation. Results: Data revealed three interrelated shifts in CRL and SRL as students progressed through clerkships. First, students’ CRL shifted from a focus on peers to co‐regulation with clinician role models. Second, self‐regulated behaviour shifted from being externally driven to being internally driven. Last, self‐regulation shifted from a task‐oriented approach towards a more comprehensive approach focusing on professional competence and identity formation. Students indicated that if they felt able to confidently and proactively self‐regulate their learning, the threshold for engaging others in meaningful CRL seemed to be lowered, enhancing further development of SRL skills. Conclusions: Findings from the current study emphasise the notion that SRL and its development are grounded in CRL in clinical settings. To optimally support the development of students’ SRL, we need to focus on facilitating and organising learners’ engagement in CRL from the start of the medical curriculum.
Processes involved in the regulation of learning have been researched for decades, because of its impact on academic and workplace performance. In fact, self‐regulated learning is the focus of countless studies in health professions education and higher education in general. While we will always need competent individuals who are able to regulate their own learning, developments in healthcare require a shift from a focus on the individual to the collective: collaboration within and between healthcare teams is at the heart of high‐quality patient care. Concepts of collaborative learning and collective competence challenge commonly held conceptualisations of regulatory learning and call for a focus on the social embeddedness of regulatory learning and processes regulating the learning of the collective. Therefore, this article questions the alignment of current conceptualisations of regulation of learning with demands for collaboration in current healthcare. We explore different conceptualisations of regulation of learning (self‐, co‐, and socially shared regulation of learning), and elaborate on how the integration of these conceptualisations adds to our understanding of regulatory learning in healthcare settings. Building on these insights, we furthermore suggest ways forward for research and educational practice.