Estimating negotiator performance without preference information

In empirical studies of negotiation support systems, it is often not possible to elicit utility functions from experimental subjects, since this would lead to undesired interventions into their behaviour. The present paper develops several methods to evaluate the performance of negotiators in multi-issue negotiations without referring to their utility functions. Data collected in negotiation experiments with the NSS Inspire are used to empirically test the match of these measures to the performance measured by utility functions elicited from the subjects. Main results of this empirical analysis are: (1) the possibilities of outside observers to evaluate the performance of negotiators without knowing their utility functions are rather limited; (2) measures using only ordinal information perform as well as measures using cardinal information; (3) taking into account observed behaviour during negotiation does not improve the fit to utilities. These results can in part be attributed to the fact that actual behaviour observed during the negotiation is incompatible with utility functions elicited in Inspire.