Reliability of data collection for the National Comprehensive Cancer Network (NCCN) non-Hodgkin lymphoma (NHL) multi-center outcomes database: A methodological approach

6525 Background: Clinical trials and outcomes studies often rely on non-physicians to abstract complex data from medical records. We assessed the reliability of chart abstraction among personnel groups in a multi-center outcomes study of indolent/aggressive NHL treated in NCCN centers. Methods: We developed 20 standardized charts of patients with newly-diagnosed NHL. Raters included 6 Clinical Research Associates from participating sites (CRAs), 3 project staff who conduct CRA training, and 3 medical oncologists. Raters each received a set of standardized charts, detailed instructions and training on a sample chart and abstracted all charts independently. We assessed reliability on 5 variables: MD-reported and rater-determined disease stage; International Prognostic Index (IPI- low-low intermediate, intermediate-high, high); Charlson comorbidity index score; and presence of any item from the Charlson index. We used intraclass correlation coefficients (ICCs) to calculate reliability. We considered coefficients from 0–0.20 ‘slight’, 0.21–0.40 ‘fair’, 0.41–0.60 ‘moderate’, 0.61–0.80 ‘substantial’ and >0.80 ‘almost perfect’(1). Results: Overall reliability was “almost perfect/substantial” for MD-reported stage, rater-determined stage, and IPI, but only “moderate” for the 2 Charlson-based comorbidity measures (Table). Reliability varied by rater group; no rater group was consistently more reliable than others. Conclusions: Trained CRAs abstracted key clinical variables with a very high degree of reliability, and performed at a level similar to study trainers and oncologists. Elements of the Charlson index were less reliable than other data types, possibly due to inherent ambiguity in the index itself. Use of trained CRA staff is reasonable to collect stage and IPI scores in a multi-center outcomes study, however abstracted Charlson scores should be interpreted with caution. (1)Biometrics. 1977. 33:159–74. No significant financial relationships to disclose. [Table: see text]