Proving the performance of a new revenue management system

A firm considering using a new revenue management (RM) model or RM system needs to evaluate its performance. In this article, we describe a ‘live’ performance measurement experiment that we conducted at Iberia Airlines on a set of flights. A set of competing algorithms control a set of flights during adjacent weeks, and their behavior and results are observed over a relatively long period of time (9 months). In parallel, a group of control flights were managed using the traditional mix of manual and algorithmic control (incumbent system). Such ‘sandbox’ testing, while common in many large internet search and e-commerce companies, is relatively rare in the RM area. Sandbox testing has an undisputable model of customer behavior but the experimental design and analysis of results is less clear. In this article, we describe the philosophy behind the experiment, the organizational challenges, the design and setup of the experiment, and outline the analysis of the results. This article is a complement to a (more technical) related paper that describes the econometrics and statistical analysis of the results.