On method overfitting

Benchmark problems should be hard. True. Methods for solving problems should be useful for more than just ‘beating’ a particular benchmark. Truer still, we believe. In this paper we examine the worthiness of the approach consisting of concentration on a particular set of benchmark problems, an issue raised by a recent paper by Ian Gent. We find that such a methodology can easily lead to publications of limited general use, as far as our ability to solve practical problems is concerned.