Summary
The gradient descent backpropagation (BP) algorithm that is widely used for training MLP neural networks can retard convergence due to certain features of the error surface like the local minimum and the flat spot. Common promoting methods, such as applying momentum term and using dynamic adaptation of learning rates, can enhance the performance of BP. However, saturation state of hidden layer neurons, which is the cause of some flat spots on the error surface, persists through such accelerating methods. In this paper, we propose a grading technique to gradually level off the potential flat spots into a sloping surface in a look-ahead mode; and thereby progressively renew saturated hidden neurons. We introduce symptoms indicating saturation state of hidden nodes. In order to suppress the saturation, we added a modifying term to the error function only when saturation is detected. In normal conditions, the improvement made to the learning process is adding a momentum term to the weight correction formula. We have recorded remarkable improvements in a selection of experiments.
See the full content of this document
Extract
Improving Backpropagation Via an Efficient Combination of a Saturation Suppression Method and Momentum Term
(ProQuest: ... denotes formulae omitted.)
1. IntroductionThe gradient descent learning procedure by error backpropagation (BP) has been the most competent multilayer neural network (MLP) training method since its initial formulation. In the error backpropagation that is an effective method to calculate the grathent of the MLP error function, the error signals (which can preserve a transformation of the weight space in summation) propagate backwards from the output nodes to the inner nodes [1-3].The error surface, as a plot of the error function against the weights, is a fine illustration of the backpropagatiou algorithm. The conventional error surface upon which the BP algorithm searches for an acceptable minimum is an ingeniously engineered and often a very hilly surface. When the MLP reaches a solution region, it has learned what is expected. The topography of the error surface, especially in complex problems, contains numbers of local minima, flat spots, and saddle points which can distort network's learning, in some cases BP may never converge. The saturation of the hidden layer neuron has been found to be a major cause of local minima in MLP trained by BP [4, 5].As a persistent problem in backpropagation algorithm, saturation of hidden neurons can form vast regions on the error surface that are nearly flat. On these flat spots, grathent assumes a very low value. As a result, before the output units have approximated to any desired signals, weight changes begin to drop to negligible amounts. Under...See the full content of this document
Sponsored links
