Neural networks: a need for caution

This paper deals with the computational aspects of neural networks. Specifically, it is suggested that the now traditional method of backpropagation (BP) may not be the most appropriate basis for learning. The argument is based on the known deficiencies of gradient descent methods, of which BP is an application. Simulation results also suggest that improved performance may be obtained by employing direct optimization procedures such as the polytope algorithm. The main reason for such performance differences appears to be that the root mean square function is subject to narrow ‘valleys’ and other anomalies.