Use and Misuse of Waterfall Plots

BACKGROUND: "Waterfall plots" are used to describe changes in tumor size observed in clinical studies. Here we assess criteria for generation of waterfall plots and the impact of measurement error in generating them. METHODS: We reviewed published waterfall plots to investigate variability in criteria used to define them. We then compared waterfall plots generated by different observers for 24 patients enrolled in a completed phase I study of solid tumors with available computed tomography (CT) scans. Tumor measurements were made independently from CT scans according to Response Evaluation Criteria in Solid Tumors 1.1 by four board-certified radiologists and four medical oncologists. Interobserver variability was quantified and compared with reference measurements reported for the phase 1 study. All statistical tests were two-sided. RESULTS: There was substantial variability in criteria used to generate published waterfall plots. In the internal study, the results were statistically significantly different between all eight readers (P = .01, variance = 197.1, SD = 14.0) and between the oncologists (P = .01, variance = 319.0, SD = 17.9), but not between the radiologists (P = .68, variance = 70.8, SD = 8.4). Different observers classified one to five patients as having a partial response and 12-19 patients as having stable disease. Similar variability in categorization of response was observed when these error rates were applied to published waterfall plots. CONCLUSION: Waterfall plots are subject to substantial variability in criteria used to define them and are influenced by measurement errors; they should be generated by trained radiologists. Caution should be exercised when interpreting results of waterfall plots in the context of clinical trials.