The University of Southampton
University of Southampton Institutional Repository

A primer for neural arithmetic logic modules

A primer for neural arithmetic logic modules
A primer for neural arithmetic logic modules
Neural Arithmetic Logic Modules have become a growing area of interest, though remain a niche field. These modules are neural networks which aim to achieve systematic generalisation in learning arithmetic and/or logic operations such as {+,−,×,÷,≤,AND} while also being interpretable. This paper is the first in discussing the current state of progress of this field, explaining key works, starting with the Neural Arithmetic Logic Unit (NALU). Focusing on the shortcomings of the NALU, we provide an in-depth analysis to reason about design choices of recent modules. A cross-comparison between modules is made on experiment setups and findings, where we highlight inconsistencies in a fundamental experiment causing the inability to directly compare across papers. To alleviate the existing inconsistencies, we create a benchmark which compares all existing arithmetic NALMs. We finish by providing a novel discussion of existing applications for NALU and research directions requiring further exploration.
arithmetic, Neural Networks, extrapolation, Interpretability, systematic generalisation
1-58
Mistry, Bhumika
36ac2f06-1a50-4c50-ab5e-a57c3faab549
Farrahi, Katayoun
bc848b9c-fc32-475c-b241-f6ade8babacb
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Mistry, Bhumika
36ac2f06-1a50-4c50-ab5e-a57c3faab549
Farrahi, Katayoun
bc848b9c-fc32-475c-b241-f6ade8babacb
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9

Mistry, Bhumika, Farrahi, Katayoun and Hare, Jonathon (2022) A primer for neural arithmetic logic modules. Journal of Machine Learning Research, 23 (185), 1-58.

Record type: Article

Abstract

Neural Arithmetic Logic Modules have become a growing area of interest, though remain a niche field. These modules are neural networks which aim to achieve systematic generalisation in learning arithmetic and/or logic operations such as {+,−,×,÷,≤,AND} while also being interpretable. This paper is the first in discussing the current state of progress of this field, explaining key works, starting with the Neural Arithmetic Logic Unit (NALU). Focusing on the shortcomings of the NALU, we provide an in-depth analysis to reason about design choices of recent modules. A cross-comparison between modules is made on experiment setups and findings, where we highlight inconsistencies in a fundamental experiment causing the inability to directly compare across papers. To alleviate the existing inconsistencies, we create a benchmark which compares all existing arithmetic NALMs. We finish by providing a novel discussion of existing applications for NALU and research directions requiring further exploration.

Text
21-0211 - Version of Record
Available under License Creative Commons Attribution.
Download (1MB)

More information

e-pub ahead of print date: 1 June 2022
Published date: 1 June 2022
Keywords: arithmetic, Neural Networks, extrapolation, Interpretability, systematic generalisation

Identifiers

Local EPrints ID: 469207
URI: http://eprints.soton.ac.uk/id/eprint/469207
PURE UUID: ef36309a-eaa0-403a-9060-b0749874854a
ORCID for Bhumika Mistry: ORCID iD orcid.org/0000-0003-4555-0121
ORCID for Katayoun Farrahi: ORCID iD orcid.org/0000-0001-6775-127X
ORCID for Jonathon Hare: ORCID iD orcid.org/0000-0003-2921-4283

Catalogue record

Date deposited: 09 Sep 2022 16:37
Last modified: 17 Mar 2024 03:58

Export record

Contributors

Author: Bhumika Mistry ORCID iD
Author: Katayoun Farrahi ORCID iD
Author: Jonathon Hare ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×