Evaluating and combining physicians’ probabilities of survival in an intensive care unit

In this paper, probabilities of survival assessed by physicians for patients admitted to an intensive care unit are studied. The probabilities from each of four types of physicians are evaluated on an overall basis and in terms of specific attributes, and the groups are compared. The physicians with the most experience and expertise perform better overall. All four groups appear to be reasonably well calibrated, and the key factor in relative overall performance is the level of discrimination provided by the probabilities. Averages of two, three, and four probabilities for each individual patient are also analyzed. As the number of the probabilities in the average increases, performance improves on average on all dimensions, although the best overall performance is exhibited by a combination of probabilities from the two physician types performing best individually. Some comparisions are made with previous work, and implications for probability assessment and combination in medicine and more generally in other areas of application are discussed. Important characteristics of the study are the fact that it was conducted on-line in a real setting, the involvement of individuals with different levels of expertise, the use of a true predictive situation with a clearly-defined event, the consideration of multiple dimensions of the quality of judgments, and the collection of multiple probabilities for each case to permit the investigation of a variety of possible combinations of probabilities.