Problem: Professionalism is a critical attribute of medical graduates. Its measurement is challenging. The authors sought to assess final-year medical students’ knowledge of appropriate professional behavior across a broad range of workplace situations. Approach: Situational judgement tests (SJTs) are used widely in applicant selection to assess judgement or decision making in work-related settings as well as attributes such as empathy, integrity, and resilience. In 2014, the authors developed three 40-item SJTs with scenarios relevant to interns (first-year junior doctors) and delivered the tests to final-year medical students to assess aspects of professionalism. As preparation, students discussed SJT-style scenarios; after the tests they completed an evaluation. The authors applied the Angoff method for the standard-setting process, delivered electronic individualized feedback reports to students post test, and provided remediation for students failing to meet the cut score. Outcomes: Evaluation revealed that the tests positively affected students’ learning and that students accepted them as an assessment tool. Validity and reliability were acceptable. Implementation costs were initially high but will be recouped over time. Next Steps: Recent improvements include changes to pass requirements, question revision based on reliability testing, and provision of detailed item-level feedback. Work is currently under way to expand the item bank and to introduce tests earlier in the course. Future research will explore correlation of SJT performance with other measures of professionalism and focus on the impact of SJTs on professionalism and interns’ ability to deal with challenging workplace situations.
12 August 2020 in Perspectives on Medical Education
Introduction The role of feedback in test-enhanced learning is an understudied area that has the potential to improve student learning. This study investigates the influence of different forms of post-test feedback on retention and transfer of biomedical knowledge within a test-enhanced learning framework. Methods 64 participants from a Canadian and an Australian medical school sat two single-best-answer formative multiple choice tests one week apart. We compared the effects of conceptually focused, response-oriented, and simple right/wrong feedback on a learner’s ability to correctly answer new (transfer) questions. On the first test occasion, participants received parent items with feedback, and then attempted items closely related (near transfer) to and more distant (far transfer) from parent items. In a repeat test at 1 week, participants were given different near and far transfer versions of parent items. Feedback type, and near and far transfer items were randomized within and across participants. Results Analysis demonstrated that response-oriented and conceptually focused feedback were superior to traditional right/wrong feedback for both types of transfer tasks and in both immediate and final retention test performance. However, there was no statistically significant difference between response-orientated and conceptually focused groups on near or far transfer problems, nor any differences in performance between our initial test occasion and the retention test 1 week later. As with most studies of transfer, participants’ far transfer scores were lower than for near transfer. Discussion Right/wrong feedback appears to have limited potential to augment test-enhanced learning.
Recent media reports have highlighted the risks to patients that may occur when practitioners in the chiropractic, osteopathy and physiotherapy professions provide services in an unethical or unsafe manner. Yet research on complaints about chiropractors, osteopaths, and physiotherapists is limited. Our aim was to understand differences in the frequency and nature of formal complaints about practitioners in these professions in order to inform improvements in professional regulation and education. This retrospective cohort study analysed all formal complaints about all registered chiropractors, osteopaths, and physiotherapists in Australia lodged with health regulators between 2011 and 2016. Based on initial assessments by regulators, complaints were classified into 11 complaint issues across three domains: performance, professional conduct, and health. Differences in complaint rate were assessed using incidence rate ratios. A multivariate negative binomial regression model was used to identify predictors of complaints among practitioners in these professions. Patients and their relatives were the most common source of complaints about chiropractors, osteopaths and physiotherapists. Concerns about professional conduct accounted for more than half of the complaints about practitioners in these three professions. Regulatory outcome of complaints differed by profession. Male practitioners, those who were older than 65 years, and those who practised in metropolitan areas were at higher risk of complaint. The overall rate of complaints was higher for chiropractors than osteopaths and physiotherapists (29 vs. 10 vs. 5 complaints per 1000 practice years respectively, p < 0.001). Among chiropractors, 1% of practitioners received more than one complaint – they accounted for 36% of the complaints within their profession. Our study demonstrates differences in the frequency of complaints by source, issue and outcome across the chiropractic, osteopathic and physiotherapy professions. Independent of profession, male sex and older age were significant risk factors for complaint in these professions. Chiropractors were at higher risk of being the subject of a complaint to their practitioner board compared with osteopaths and physiotherapists. These findings may assist regulatory boards, professional associations and universities in developing programs that avert patient dissatisfaction and harm and reduce the burden of complaints on practitioners.
Programmatic assessment (PA) has strong theoretical and pedagogical underpinnings, but its practical implementation brings a number of challenges—particularly in traditional university settings involving large cohort sizes. This paper presents a detailed case report of an in-progress programmatic assessment implementation involving a decade of assessment innovation occurring in three significant and transformative steps. The starting position and subsequent changes represented in each step are reflected against the framework of established principles and implementation themes of PA. This case report emphasises the importance of ongoing innovation and evaluative research, the advantage of a dedicated team with a cohesive plan, and the fundamental necessity of electronic data collection. It also highlights the challenge of traditional university cultures, the potential advantage of a major pandemic disruption, and the necessity for curriculum renewal to support significant assessment change. Our PA implementation began with a plan to improve the learning potential of individual assessments and over the subsequent decade expanded to encompass a cohesive and course wide assessment program involving meaningful aggregation of assessment data. In our context (large cohort sizes and university-wide assessment policy) regular progress review meetings and progress decisions based on aggregated qualitative and quantitative data (rather than assessment format) remain local challenges.