Beyond validation: assessing the legitimacy of artificial neural network models

Abstract

Artificial neural network models have been used extensively for prediction and forecasting over the last 25 years. As the data used to develop ANNs contain important information about the physical processes being modelled, it is generally implied that a model that has been calibrated (trained) and performs well on an independent set of validation data represents the underlying physical processes of the system being modelled. However, this is not necessarily the case, most likely due to problems with equifinality, where different combinations of model parameters (e.g. connection weights) result in similar predictive performance. Consequently, there is also a need to check the behaviour of calibrated ANN models as part of the validation process, which is commonly referred to as structural, conceptual or scientific validation (Figure 1). This checks whether the input-output relationship captured by the model is plausible in accordance with a priori system understanding. Figure 1. Importance of checking both predictive accuracy and model behaviour during ANN model validation processes In this paper, the importance of considering structural validation is demonstrated. This is achieved by developing ANN models with different numbers of hidden nodes for two environmental modelling case studies from the literature namely, salinity forecasting in the River Murray in South Australia and the prediction of treated water turbidity at a water treatment plant based on raw water quality and the administered alum dose. The validation errors are then compared with corresponding model behaviours. This was done using the validann R-package, which caters to a range of structural validation approaches. Results show that ANN models producing the best fit to the data do not necessarily result in models that behave in accordance with underlying system understanding.