Evaluating the reliability of a tool to measure the quality of gastrointestinal multidisciplinary cancer conferences: A generalizability study

Background Lamb et al. developed the metric for the observation of decision-making tool (MTB-MODe) to evaluate the quality of urologic multidisciplinary cancer conferences (MCCs) in the United Kingdom. We used generalizability theory to assess the reliability of a modified version of MTB-MODe in a North American context. Specifically, we wished to determine if the tool could distinguish between high- and low-quality MCC decision-making. Methods Two assessors independently evaluated two MCCs (MCC1, MCC2) using the modified MTB-MODe. Generalizability theory was used to assess overall tool reliability and to identify sources most likely to contribute to variance in reliability scores. A total of 60 cases were evaluated. Results The overall reliability scores of MCC1 and MCC2 were 0.72 and 0.74, respectively. Inter-rater reliability scores were reasonable (>0.55) and raters did not contribute significantly to variance in reliability scores. Internal consistency of the individual MTB-MODe items was low, demonstrating that items were not highly correlated. Conclusions The MTB-MODe reliably assessed the quality of individual MCC cases. Raters did not contribute significantly to reliability scores, suggesting that the tool can be successfully implemented using a single rater. Low internal consistency of the MTB-MODe items demonstrates that the tool can be used to provide feedback on individual tool items. Such data can be used by stakeholders to help improve MCC quality.