The Canadian ophthalmology assessment tool for surgery: a surgical assessment tool for ophthalmology training programs

OBJECTIVE: To develop a surgical assessment tool to evaluate the breadth of surgical procedures in which ophthalmology residents are required to gain competence and to validate this tool for cataract surgery. DESIGN: A questionnaire development study. PARTICIPANTS: Ophthalmology residents and faculty at the University of Ottawa. METHODS: The Canadian Ophthalmology Assessment Tool for Surgery (COATS) was developed by a group of experts by modifying a previously validated assessment tool, the Ottawa Surgical Competency Operating Room Evaluation, and consists of 6 Likert scale items, 1 yes or no question assessing independence and 2 open-ended questions for narrative feedback. During a 2-year period, residents were evaluated on the performance of cataract surgery using the COATS. The primary outcome was the total COATS score, defined as the sum of the Likert scale items. RESULTS: A total of 165 COATS assessments were collected across 5 residents. Mean total COATS scores were lower for the first 2 months of training compared with the last 3 months of training (4.33 ± 0.25 vs 4.81 ± 0.03; p = 0.01; Cohen's d = 0.25) and for procedures where the resident was rated as "not independent" versus "independent" (4.26 ± 0.13 vs 4.74 ± 0.06; p = 0.006; Cohen's d = 0.13). There was a significant correlation between the number of cataract surgeries performed and the mean total COATS score (Pearson r = 0.20; p = 0.02). Forty-five COATS assessments per resident were required to obtain an overall reliability of 0.70, the accepted threshold for low-stake assessments. CONCLUSIONS: The COATS is a valid tool for the assessment of surgical competence in cataract surgery. There is also evidence that it is a reliable tool when completed multiple times per resident over the course of training.