The Reliability of Horizontal Assessment for...

The Reliability of Horizontal Assessment for Scoring Physiotherapy Entrance Interviews and Optimizing Rater Allocation

Abstract

Introduction. This study estimated the interrater reliability of a fully horizontal admissions review of physiotherapy candidates and compared it with previous ratings of partial horizontal and vertical reviews. In addition, a method for optimizing the allocation of admissions rater resources to enhance reliability is presented. Review of the Literature. Previous investigations have determined that admissions interview reviews structured in a partial horizontal format reduced bias and increased reliability when compared to vertical reviews. Subjects. A Canadian physiotherapy program admissions reviewers’ scoring data from 2021 to 2023 were included. Methods. This quality improvement initiative evaluated retrospective data from asynchronous virtual Multiple-Mini Interviews (AVMMI) to estimate interrater reliability and measurement error of a fully horizontal scored 2023 applicant cohort, compared to previously reported 2020 vertical, and 2021 partially horizontal scored interviews. Shrout and Fleiss Class 1,2 intraclass correlation coefficients (ICC) and standard errors of measurement (SEM) were calculated for the average of 2-raters’ scores. The second component involved creating generalizability models that identified variance components related to candidate and rater variables, including the number of raters and questions. Results. The 2023 interrater reliability for the mean of 2-raters was 0.82 compared to 0.63 for the 2020 completely vertical and 0.75 for the partially horizontal historical datasets. The estimated variance components and their adjustment to each of the 3 conditions: 2-raters/question, 7-questions; 1-rater/question, 8-questions; 2-raters/question, 4-questions revealed the following: the largest generalizability coefficient (G) and smallest SEM were obtained for the current interview format of 2-raters/question and 7-questions. Discussion and Conclusions. The horizontal admissions review increased rater reliability and decreased error variance compared with vertical and partial horizontal reviews. This analysis illustrated that when rater resources are scarce, it is better to increase the number of interview questions and use a separate rater per question versus reducing the number of questions and maintaining 2 raters per question.