Foveated convolutions: improving spatial transformer networks by modelling the retina

Abstract

Spatial Transformer Networks (STNs) have the potential to dramatically improve performance of convolutional neural networks in a range of tasks. By ‘focusing’ on the salient parts of the input using a differentiable affine transform, a network augmented with an STN should have increased performance, efficiency and interpretability. However, in practice, STNs rarely exhibit these desiderata, instead converging to a seemingly meaningless transformation of the input. We demonstrate and characterise this localisation problem as deriving from the spatial invariance of feature detection layers acting on extracted glimpses. Drawing on the neuroanatomy of the human eye we then motivate a solution: foveated convolutions. These parallel convolutions with a range of strides and dilations introduce specific translational variance into the model. In so doing, the foveated convolution presents an inductive bias, encouraging the subject of interest to be centred in the output of the attention mechanism, giving significantly improved performance.

More information

Identifiers

ORCID for Ethan William Albert Harris: orcid.org/0000-0003-3545-1349

ORCID for Mahesan Niranjan: orcid.org/0000-0001-7021-140X

ORCID for Jonathon Hare: orcid.org/0000-0003-2921-4283

Catalogue record

Export record