Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED. The resuscitation assessment tool (RAT) was derived from the previously implemented and studied Queen’s simulation assessment tool (QSAT) via a modified expert review process. The RAT uses an anchored global assessment scale to generate an entrustment score and narrative comments. Emergency medicine (EM) residents were assessed using the RAT on cases in simulation-based examinations and in the ED during resuscitation cases from July 2016 to June 2017. Resident mean entrustment scores were compared using Pearson’s correlation coefficient to determine the relationship between entrustment in simulation cases and in the ED. Inductive thematic analysis of written commentary was conducted to compare workplace-based with simulation-based feedback. There was a moderate, positive correlation found between mean entrustment scores in the simulated and workplace-based settings, which was statistically significant (r = 0.630, n = 17, p < 0.01). Further, qualitative analysis demonstrated overall management and leadership themes were more common narratives in the workplace, while more specific task-based feedback predominated in the simulation-based assessment. Both workplace-based and simulation-based narratives frequently commented on communication skills. In this single-center study with a limited sample size, assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace. This study suggests that resuscitation performance in simulation settings may be an indicator of competence in the clinical setting. However, multiple factors contribute to this complicated and imperfect relationship. It is imperative to consider narrative comments in supporting the rationale for numerical entrustment scores in both settings and to include both simulation and workplace-based assessment in high-stakes decisions of progression.

Problem: Assessing the development and achievement of competence requires multiple formative and summative assessment strategies and the coordinated efforts of trainees and faculty (who often serve in multiple roles, such as academic advisors, program directors, and competency committee members). Operationalizing programmatic assessment (PA) in competency-based medical education (CBME) requires comprehensive practice guidelines, written in accessible language with descriptions of stakeholder activities, to move assessment theory into practice and to help guide the trainees and faculty who enact PA. Approach: Informed by the Appraisal of Guidelines for Research and Evaluation II (AGREE II) framework, the authors used a multiphase, multimethod approach to develop the CBME Programmatic Assessment Practice Guidelines (PA Guidelines). The 9 guidelines are organized by phases of assessment and include descriptions of stakeholder activities. A user guide provides a glossary of key terms and summarizes how the guidelines can be used by different stakeholder groups across postgraduate medical education (PGME) contexts. The 4 phases of guideline development, including internal stakeholder consultations and external expert review, occurred between August 2016 and March 2020. Outcomes: Local stakeholders and external experts agreed that the PA Guidelines hold potential for guiding initial operationalization and ongoing refinement of PA in CBME by individual stakeholders, residency programs, and PGME institutions. Since July 2020, the PA Guidelines have been used at Queen’s University to inform faculty and resident development initiatives, including online CBME modules for faculty, workshops for academic advisors/competence committee members, and a guide that supports incoming residents’ transition to CBME. Next Steps: Research exploring the use of the PA Guidelines and user guide in multiple programs and institutions will gather further evidence of their acceptability and utility for guiding operationalization of PA in different contexts.

Professional accrediting and regulating bodies are increasingly trying to delineate the knowledge and skills needed for entry-to-practice for quality assurance and international labor mobility. The purpose of this study was to com-pare how professions describe and represent competence. Current, publicly accessible Canadian entry-to-practice competence frameworks for ten professions (Medicine, Nursing, Occupational Therapy, Pharmacy, Psychology, Social Work, Teaching, Engineering, Law, and Planning) were analyzed through content and inductive thematic analysis. Findings revealed that although professions describe similar core competencies across technical and intrinsic domains, systematic differences exist in the architectures of the frameworks. Professions with ‘meta-competencies’ describe competence as being more integrated/holistic. Whereas professions without meta-competencies describe competence as either behavioral-/performance-like lists of ‘attributes’ or groups of knowledge, skills, and ethical/professional values. How competence is represented within frameworks has implications for how professional education programs conceptualize competence and subsequently design and enact curricula, teaching and learning opportunities, and systems of assessment.

While most case studies consider how programs of assessment may influence residents’ achievement, we engaged in a qualitative, multiple case study to model how resident engagement and performance can reciprocally influence the program of assessment. We conducted virtual focus groups with program leaders from four residency training programs from different disciplines (internal medicine, emergency medicine, neurology, and rheumatology) and institutions. We facilitated discussion with live screen-sharing to (1) improve upon a previously-derived model of programmatic assessment and (2) explore how different resident archetypes (sample profiles) may influence their program of assessment. Participants agreed that differences in resident engagement and performance can influence their programs of assessment in some (mal)adaptive ways. For residents who are disengaged and weakly performing (of which there are a few), significantly more time is spent to make sense of problematic evidence, arrive at a decision, and generate recommendations. Whereas for residents who are engaged and performing strongly (the vast majority), significantly less effort is thought to be spent on discussion and formalized recommendations. These findings motivate us to fulfill the potential of programmatic assessment by more intentionally and strategically challenging those who are engaged and strongly performing, and by anticipating ways that weakly performing residents may strain existing processes.

Time-honoured university policies, such as the credit-hour and academic freedom, present challenges for professional education programs tasked with operationalizing entry-to-practice competence frameworks for professional accreditation. A single embedded case study was used to explore how professional programs from one mid-sized Canadian university are approaching and perhaps problematizing the development and assessment of competence. Semi-structured interviews were conducted with educational leaders (faculty and staff, n=21) from a sample of nine programs. Following a grounded theory approach to qualitative analysis, the constant comparative method was used to inductively discern similarities and differences across programs, and to begin building theory about approaches to operationalization. While limited in scope given the use of a single university, our findings highlight: (a) diversity in approaches to operationalization across programs, (b) common attributes which can be used to classify the manner in which these programs operationalize competence, and (c) challenges with supporting faculty to buy in to competency-informed pedagogy and assessment. Given these findings, it is recommended that professional accrediting bodies and education programs spend time to consider the role university-based programs play in determining competence for entry-to-practice, as well their intents for implementing a competence framework, to ensure sufficiency in the approaches being used.