Evaluation of confidence intervals for the kappa statistic when the assumption of marginal homogeneity is violated

This article studies the robustness of confidence interval construction for the intraclass kappa statistic based on a dichotomous response when the assumption of marginal homogeneity across two raters is violated. Two methods of construction are considered: the goodness-of-fit approach and the modified Wald method. Evaluation was done by exact calculation of the confidence interval coverage produced by these approaches. It was found that under mild departures from marginal homogeneity (differences in rater success rates of <$$<$$10 %), the goodness- of-fit approach can be recommended. Moreover, under these same conditions, Cohen’s kappa tends to be less biased as a point estimator than the intraclass kappa statistic.