Article ID: | iaor201524973 |
Volume: | 55 |
Issue: | 3 |
Start Page Number: | 221 |
End Page Number: | 234 |
Publication Date: | Sep 2013 |
Journal: | Australian & New Zealand Journal of Statistics |
Authors: | Abeysekera Waruni, Kabaila Paul, Yilmaz Oguzhan |
Keywords: | simulation |
Volume 3 of Analysis of Messy Data by Milliken & Johnson (2002) provides detailed recommendations about sequential model development for the analysis of covariance. In his review of this volume, Koehler (2002) asks whether users should be concerned about the effect of this sequential model development on the coverage probabilities of confidence intervals for comparing treatments. We present a general methodology for the examination of these coverage probabilities in the context of the two‐stage model selection procedure that uses two F tests and is proposed in Chapter 2 of Milliken & Johnson (2002). We apply this methodology to an illustrative example from this volume and show that these coverage probabilities are typically very far below nominal. Our conclusion is that users should be very concerned about the coverage probabilities of confidence intervals for comparing treatments constructed after this two‐stage model selection procedure.