Efficacy of end-user neural network and data mining software for predicting complex system performance

The performance of a university's dial-up modem pool under various time limit policies and customer behavior patterns was studied. Because the system is very complex, simulation offered the only method to obtain a limited set of steady-state performance measure estimates. A more generalized predictive model must be built from the simulated output. Traditional methods available to practitioners for predicting system performance across a range of environmental and decision variables have typically been limited to linear regression models. However, when the system being studied is highly complex and its performance is nonlinear in nature, the effectiveness of linear models can be limited. While more advanced nonlinear methods, such as neural networks, have been shown to perform better than traditional regression analysis in these situations, the knowledge needed to implement them ‘from scratch’ is beyond most practitioners. Fortunately, these advanced methods are now available in ready-to-use desktop software programs, making them more attainable for practitioner use. The efficacy of these end-user programs compared to more traditional methods in practice is of interest. Multiple variable linear regression models were developed for predicting six output measures in a simulation study and were compared to nonlinear regression models developed using a data mining software package and two commerial neural network software packages. Comparisons of the models' predictive ability were made on both the data used to design the models and on a test set of data. Statistical analysis shows that predictive performance on the test data was usually best with one of the neural network models, but relative performance of the different models varied widely.