The aim of this study was to assess validation evidence for a sedation scale for dogs. We hypothesized that the chosen sedation scale would be unreliable when used by different raters and show poor discrimination between sedation protocols. A sedation scale (range 0–21) was used to score 62 dogs scheduled to receive sedation at two veterinary clinics in a prospective trial. Scores recorded by a single observer were used to assess internal consistency and construct validity of the scores. To assess inter-rater reliability, video-recordings of sedation assessment were randomized and blinded for viewing by 5 raters untrained in the scale. Videos were also edited to allow assessment of inter-rater reliability of an abbreviated scale (range 0–12) by 5 different raters. Both sedation scales exhibited excellent internal consistency and very good inter-rater reliability (full scale, intraclass correlation coefficient [ICCsingle] = 0.95; abbreviated scale, ICCsingle = 0.94). The full scale discriminated between the most common protocols: dexmedetomidine-hydromorphone (median [range] of sedation score, 11 [1–18], n = 20) and acepromazine-hydromorphone (5 [0–15], n = 36, p = 0.02). The hypothesis was rejected. Full and abbreviated scales showed excellent internal consistency and very good reliability between multiple untrained raters. The full scale differentiated between levels of sedation.

Veterinary medical faculty and administrators routinely administer student assessments and conduct surveys to make decisions regarding student performance and to assess their courses/curricula. The decisions that are made are a result of the scores generated. However, how reliable are the scores and how confident can we be about these decisions? Reliability is one of the hallmarks of validity evidence, but what does this mean and what affects the reliability of scores? The purpose of this article is to provide veterinary medical educators and administrators with fundamental information regarding the concept of reliability. Specifically, we review what sources of error reduce the reliability of scores and we describe the different types of reliability coefficients that are reported.

Background: Medical school curricula have undergone considerable change in the past half century. There is little evidence, however, for the impact of various curricula and educational policies on student learning once incoming performance and the nonrandom nature of students nested within schools has been accounted for. Purpose: To investigate effects of school variables on United States Medical Licensing Examination (USMLE) Step 1–3 scores over an 11-year period (1994–2004). Methods: Using Association of American Medical Colleges and USMLE longitudinal data for 116 medical schools, hierarchical linear modeling was used to study the effects of school variables on Step 1–3. Results: Mean unadjusted between school variance was 14.74%, 10.50%, and 11.25%, for USMLE Step 1–3. When student covariates were included, between-school variation was less than 5%. The proportion of variance accounted for in-student-level performance by the covariates ranged from 27.58% to 36.51% for Step 1, 16.37% to 24.48% for Step 2, and 19.22% to 25.32% for Step 3. The variance accounted for in student performance by the student covariates ranged between 81.22% and 88.26% for Step 1, 48.44% and 79.77% for Step 2, and 68.41% and 80.78% for Step 3. School-level variables did not consistently predict for adjusted mean school Step performance. Conclusions: Individual student differences account for most of the variation in USMLE performance with small contributions from between-school variation and even smaller contribution from curriculum and educational policies.

This study describes the development, implementation, and psychometric assessment of the multiple mini-interview (MMI) for the inaugural class of veterinary medicine applicants at the University of Calgary Faculty of Veterinary Medicine (UCVM). The MMI is a series of approximately five to 12 10-minute interviews that consist of situational events. Applicants are given a scenario and asked to work through an issue or behavioral-type questions that are meant to assess one attribute (e.g., empathy) at a time. This structure allows for multiple assessments of the applicants by trained interviewers on the same questions. MMI scenario development was based on a review of the noncognitive attributes currently assessed by the 31 veterinary schools across Canada and the United States and the goals and objectives of UCVM. The noncognitive attributes of applicants (N = 110) were assessed at five stations, by two interviewers within each station, on three items using a standardized rating form on an anchored 1–5 scale. The method was determined to be reliable (G-coefficient = 0.88) and demonstrated evidence of validity. The MMI score did not correlate with grade-point average (r = 0.12, p = 0.22). While neither the applicants nor interviewers had participated in an MMI format before, both groups reported the process to be acceptable in a post-interview questionnaire. This analysis provides preliminary evidence of the reliability, validity, and acceptability of the MMI in assessing the noncognitive attributes of applicants for veterinary medical school admissions.