Analytic choices in road safety evaluation: Exploring second‐best approaches

Conducting rigorous before‐and‐after studies is essential for improving knowledge regarding the effects of road safety measures. However, state‐of‐the‐art approaches like the empirical Bayes or fully Bayesian techniques cannot always be applied, as the data required by these approaches may be missing or unreliable. The choice facing researchers in such a situation is to either apply ‘second‐best’ approaches or abstain from doing an evaluation study. An objection to applying second‐best approaches is that these approaches do not control as well for confounding factors as state‐of‐the‐art approaches. This paper explores the implications of choice of study design by examining how the findings of several evaluation studies made in Norway depend on choices made with respect to:

Using the empirical Bayes approach versus using simpler approaches;

Use or non‐use of a comparison group;

The choice of comparison group when there is more than one candidate.

It is found that the choices made with respect to these points can greatly influence the estimates of safety effects in before‐and‐after studies. Two second‐best techniques (i.e. techniques other than the empirical Bayes approach) for controlling for confounding factors were tested. The techniques were found not to produce unbiased estimates of effect and their use is therefore discouraged.