Outcome assessment for clinical trials: How many adjudicators do we need?

Considerable effort is often expended to adjudicate outcomes in clinical trials, but little has been written on the administration of the adjudication process and its possible impact on study results. As a case study, we describe the function and performance of an adjudication committee in a large randomized trial of two diagnostic approaches to potentially operable lung cancer. Up to five independent adjudicators independently determined two primary outcomes: tumor status at death or at final follow-up and the cause of death. Patients for whom there was any disagreement were discussed in committee until a consensus was achieved. We describe the pattern of agreement among the adjudicators and with the final consensus result. Additionally, we model the adjudication process and predict the results if a smaller committee had been used. We found that reducing the number of adjudicators from five to two or three would probably have changed the consensus outcome in less than 10% of cases. Correspondingly, the effect on the final study results (comparing primary outcomes in both randomized arms) would have been altered very little. Even using a single adjudicator would not have affected the results substantially. About 90 minutes of person-time per patient was required for activities directly related to the adjudication process, or approximately 6 months of full time work for the entire study. This level of effort could be substantially reduced by using fewer adjudicators with little impact on the results. Thus, we suggest that when high observer agreement is demonstrated or anticipated, adjudication committees should consist of no more than three members. Further work is needed to evaluate if smaller committees are adequate to detect small but important treatment effects or if they compromise validity when the level of adjudicator agreement is lower.