The University of Southampton
University of Southampton Institutional Repository

A Global Gradient Noise Covariance Expression for Stationary Real Gaussian Inputs

A Global Gradient Noise Covariance Expression for Stationary Real Gaussian Inputs
A Global Gradient Noise Covariance Expression for Stationary Real Gaussian Inputs
Supervised parameter adaptation in many artificial neural networks is largely based on an instantaneous version of gradient descent called the Least-Mean-Square (LMS) algorithm. As the gradient is estimated using single samples of the input ensemble, its convergence properties generally deviate significantly from that of the true gradient descent because of the noise in the gradient estimate. It is thus important to study the gradient noise characteristics so that the convergence of the LMS algorithm can be analyzed from a new perspective. This paper considers only neural models which are linear with respect to their adaptable parameters, and has two major contributions. Firstly, it derives an expression for the gradient noise covariance under the assumption that the input samples are real, stationary, Gaussian distributed but can be partially correlated. This expression relates the gradient correlation and input correlation matrices to the gradient noise covariance, and explains why the gradient noise generally correlates maximally with the steepest principal axis and minimally with the one of the smallest curvature, regardless of the magnitude of the weight error. Secondly, a recursive expression for the weight error correlation matrix is derived in a straightforward manner using the gradient noise covariance, and comparisons are drawn with the complex LMS algorithm.
An, P.E.
5dc94657-d009-4d13-9a0f-6645a9d296d9
Brown, M.
52cf4f52-6839-4658-8cc5-ec51da626049
Harris, C.J.
c4fd3763-7b3f-4db1-9ca3-5501080f797a
An, P.E.
5dc94657-d009-4d13-9a0f-6645a9d296d9
Brown, M.
52cf4f52-6839-4658-8cc5-ec51da626049
Harris, C.J.
c4fd3763-7b3f-4db1-9ca3-5501080f797a

An, P.E., Brown, M. and Harris, C.J. (1995) A Global Gradient Noise Covariance Expression for Stationary Real Gaussian Inputs. IEEE Trans. on Neural Networks, 6 (6).

Record type: Article

Abstract

Supervised parameter adaptation in many artificial neural networks is largely based on an instantaneous version of gradient descent called the Least-Mean-Square (LMS) algorithm. As the gradient is estimated using single samples of the input ensemble, its convergence properties generally deviate significantly from that of the true gradient descent because of the noise in the gradient estimate. It is thus important to study the gradient noise characteristics so that the convergence of the LMS algorithm can be analyzed from a new perspective. This paper considers only neural models which are linear with respect to their adaptable parameters, and has two major contributions. Firstly, it derives an expression for the gradient noise covariance under the assumption that the input samples are real, stationary, Gaussian distributed but can be partially correlated. This expression relates the gradient correlation and input correlation matrices to the gradient noise covariance, and explains why the gradient noise generally correlates maximally with the steepest principal axis and minimally with the one of the smallest curvature, regardless of the magnitude of the weight error. Secondly, a recursive expression for the weight error correlation matrix is derived in a straightforward manner using the gradient noise covariance, and comparisons are drawn with the complex LMS algorithm.

This record has no associated files available for download.

More information

Published date: 1995
Additional Information: accepted for publication
Organisations: Southampton Wireless Group

Identifiers

Local EPrints ID: 250282
URI: http://eprints.soton.ac.uk/id/eprint/250282
PURE UUID: 87d7d06d-7b40-450b-979e-6d710033f65c

Catalogue record

Date deposited: 04 May 1999
Last modified: 10 Dec 2021 20:07

Export record

Contributors

Author: P.E. An
Author: M. Brown
Author: C.J. Harris

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×