Clinician’s ability to identify neck and low back interventions: an inter-rater chance-corrected agreement pilot study

OBJECTIVE: To estimate inter-rater agreement of physical therapists trained in MDT approach and participating in practice-based evidence (PBE) research to identify 72 physical therapy interventions in video demonstrations on a single model and clinical vignettes. PBE is a well designed observational study and demonstrating clinician observational consistency is an important step in conducting PBE research design. METHODS: Two physical therapists volunteered to participate in pilot reliability testing and seven other physical therapists trained in McKenzie Mechanical Diagnosis and Therapy (MDT) methods volunteered for the inter-rater chance-corrected agreement study. All therapists identified interventions presented within 52 videos and 5 written clinical vignettes describing 20 more intervention techniques. Therapists independently identified all interventions. We assessed inter-rater chance-corrected agreement of therapists' ability to identify intervention techniques using Kappa coefficients with associated 95% confidence intervals and indices for bias and prevalence. RESULTS: Of the 147 kappa coefficients estimated, 7% were ⩽0·6, 10% were >0·6 and ⩽0·8, and 83% were >0·8. Agreement was lowest for identifying cognitive behavioral techniques (median kappa = 0·79). The minimum and maximum prevalence and bias indices were 0·33 and 0·85 and 0 and 0·33, respectively suggesting kappa coefficient estimates were strong. Generalized kappa coefficients ranged from 0·73 to 1·00. DISCUSSION: Results provide evidence that substantial to almost perfect inter-rater agreement could be expected when trained therapists identify physical therapy interventions used for patients with spinal impairments from staged videos and vignettes. This may be helpful to reassure clinicians of the quality of the reporting of intervention(s) performed when conducting multivariable analyses in future pragmatic PBE studies. Additional studies are needed to test whether these results can be validated using larger groups of therapists, trained and not trained in MDT methods, as well as examining different methods to examine inter-rater agreement for identifying diverse interventions commonly used for managing patients during routine practice.