The False Promise of Big Data: Can Data Mining Replace Hypothesis-Driven Learning in the Identification of Predictive Performance Metrics?

This paper argues US manufacturers still fail to identify metrics that predict performance results despite two decades of intensive investment in data‐mining applications because indicators with the power to predict complex results must have high information content as well as a high impact on those results. But data mining cannot substitute for experimental hypothesis testing in the search for predictive metrics with high information content–not even in the aggregate–because the low‐information metrics it provides require improbably complex theories to explain complex results. So theories should always be simple but predictive factors may need to be complex. This means the widespread belief that data mining can help managers find prescriptions for success is a fantasy. Instead of trying to substitute data mining for experimental hypothesis testing, managers confronted with complex results should lay out specific strategies, test them, adapt them–and repeat the process.