Why is performance on multiple-choice tests and constructed-response tests not more closely related? Theory and an empirical test

Both professional certification and academic tests rely heavily on multiple-choice questions, despite the widespread belief that alternate, constructed-response questions are superior measures of a test taker's understanding of the underlying material. Empirically, the search for a link between these two assessment metrics has met with limited success, leading some researchers to conclude that the relationship is close and others to conclude that no relationship exists at all. The authors suggest that ‘knowledge level’ may play a key role in explaining this disparity in findings. This article outlines the theory for such a concept, and investigates the possibility using 172 carefully constructed tests in several entry-level programming classes. The article also discusses several caveats that suggest the usefulness of yet further research in the area.