Neural networks and finite-order approximations

This paper investigates the approximation properties of standard feedforward neural networks (NNs) through the application of multivariate Taylor-series expansions. The capacity to approximate arbitrary functional forms is central to the NN philosophy, but is usually proved by allowing the number of hidden nodes to increase to infinity. The Taylor-series approach does not depend on such limiting cases. The paper shows how the series approximation depends on individual network weights. The role of the bias term is taken as an example. We are also able to compare the sigmoid and hyperbolic-tangent activation functions, with particular emphasis on their impact on the bias term. The paper concludes by discussing the potential importance of our results for NN modelling: of particular importance is the training process.