ABSTRACT

When evaluating the usefulness of clinical information for the diagnosis of disease, multiple raters provide a diagnosis for the same set of data. These ratings provide important insights into the performance of the diagnosis, determining the accuracy of each rater’s diagnosis compared to the truth standard and the level of agreement among the raters. We demonstrate that the intraclass correlation coefficient (ICC) is dependent on the sensitivities and specificities of the raters involved in the study. Given the sensitivity and specificity of any number of raters, along with the prevalence of disease, the expected ICC can be determined.