Summary
This paper focuses on gradient-based backpropagation algorithms that use either a common adaptive learning rate for all weights or a separate adaptive learning rate for each weight. The learning-rate adaptation is based on descent techniques and estimates of the local constants that are obtained without additional error function and gradient evaluations. This paper proposes three algorithms to improve the different versions of backpropagation training in terms of both convergence rate and convergence characteristics, such as stable learning and robustness to oscillations. The new modification consists of a simple change in the error signal function. Experiments are conducted to compare and evaluate the convergence behavior of these gradient-based training algorithms with three training problems: XOR, encoding problem and character recognition, which are popular training problems.
See the full content of this document
Extract
Efficient Training of Backpropagation Neural Networks
1. Introduction
The Backpropagation (BP) algorithm [22, 23] is perhaps the most widely used supervised training algorithm for multi- layered feed- forward neural networks. However, in some cases, the standard Backpropagation takes unendurable time to adapt the weights between the units in the network to minimize the mean squared errors between the desired outputs and the actual network outputs [4] .A variety of approaches adapted from the numerical analysis have been applied in an attempt to use not only the gradient of the error function but also the second derivative in constructing efficient supervised training algorithms to accelerate the learning process. The research usually focuses on heuristic methods for dynamically adapting the learning rate during training to accelerate the convergence.There are many researches, which have been proposed to improve this algorithm; some of these researches were developed to speeding up a training process [5, 8, 12, 13, 26, 28] . Other researches have investigated the effect of adaption of momentum factor [14, 29, 30].This paper presents three versions of an optical backpropagation (OBP) algorithm, with analysis of its benefits. An OBP algorithm is designed to overcome some of the problems associated with standard BP training using the non- linear function, which applied on the output units. One of the most important aspects of the proposed algorithm is its abi...See the full content of this document
Sponsored links
