| Article ID: | iaor20081859 |
| Country: | Netherlands |
| Volume: | 42 |
| Issue: | 4 |
| Start Page Number: | 2236 |
| End Page Number: | 2246 |
| Publication Date: | Jan 2007 |
| Journal: | Decision Support Systems |
| Authors: | Chen Hsinchun, Schumacher Robert P., Ginsburg Mark, Liu Ling |
An effective networked knowledge delivery platform is one of the Holy Grails of Web computing. Knowledge delivery approaches range from the heavy and narrow to the light and broad. This paper explores a lightweight and flexible dialog framework based on the ALICE system, and evaluates its performance in chat and knowledge delivery using both a conversational setting and a specific telecommunications knowledge domain. Metrics for evaluation are presented, and the evaluations of three experimental systems (a pure dialog system, a domain knowledge system, and a hybrid system combining dialog and domain knowledge) are presented and discussed. Our study of 257 subjects shows approximately a 20% user correction rate on system responses. Certain error classes (such as nonsense replies) were particular to the dialog system, while others (such as mistaking opinion questions for definition questions) were particular to the domain system. A third type of error, wordy and awkward responses, is a basic system property and spans all three experimental systems. We also show that the highest response satisfaction results are obtained when coupling domain-specific knowledge together with conversational dialog.