Quantifying the Random Component of Measurement Error of Nominal Measurements Without a Gold Standard

It is well known that measurement error of numerical measurements can be divided into a systematic and a random component and that only the latter component is estimable if there is no gold standard or reference standard available. In this paper, we consider measurement error of nominal measurements. We motivate that, on a nominal measurement scale too, measurement error has a systematic and a random component and only the random component is estimable without gold standard. Especially in literature about binary measurement error, it is common to quantify measurement error by ‘false classification probabilities’: the probabilities that measurement outcomes are unequal to the correct outcomes. These probabilities can be split up in a systematic and a random component. We quantify the random component by ‘inconsistent classification probabilities’ (ICPs): the probabilities that a measurement outcome is unequal to the modal (instead of correct) outcome. Systematic measurement error then is the event that this modal outcome is unequal to the correct outcome. We introduce an estimator for the ICPs and evaluate its properties in a simulation study. We end with a case study that demonstrates not only the determination and use of the ICPs but also demonstrates how the proposed modeling can be used for formal hypothesis testing. Things to test include differences between appraisers and random classification by a single appraiser.