When Less Is More: Data and Power in Advertising Experiments

Yahoo! Research partnered with a nationwide retailer to study the effects of online display advertising on both online and in‐store purchases. We use a randomized field experiment on 3 million Yahoo! users who are also past customers of the retailer. We find statistically significant evidence that the retailer ads increase sales 3.6% relative to the control group. We show that control ads boost measurement precision by identifying and removing the half of in‐campaign sales data that are unaffected by the ads. Less data give us 31% more precision in our estimates–equivalent to increasing our sample to 5.3 million users. By contrast, we only improve precision by 5% when we include additional covariate data to reduce the residual variance in our experimental regression. The covariate‐adjustment strategy disappoints despite exceptional consumer‐level data including demographics, ad exposure levels, and two years’ worth of past purchase history. Data, as supplemental material, are available at http://dx.doi.org/10.1287/mksc.2016.0998.