An investigation into neural arithmetic logic modules

Abstract

The human ability to learn and reuse skills in a systematic manner is critical to our daily routines. For example, having the skills for executing the basic arithmetic operations (+,−,×,÷) allows a person to perform a variety of tasks including budgeting expenses, scaling measurements to the desired proportions when cooking/baking, and planning travel schedules. Machine Learning (ML) can reduce the manual workload for humans, inferring underlying relations within the data without the need for heavy feature engineering. However, the ability of such models to extrapolate and generalise to unseen data in an interpretable manner is challenging. With this challenge in mind, Neural Arithmetic Logic Modules (NALMs) have been developed. Such parameterised modules, specialised for arithmetic operations, are designed to guarantee generalisation if weights are correctly learned and be interpretable in what they learn. This thesis seeks to thoroughly investigate the proposition that such specialised differentiable modules with inductive biases toward arithmetic can be learned, uncovering the limitations which remain. In this work, we begin by studying the extent to which NALMs are able to learn arithmetic. We initially provide a comprehensive review of existing NALMs and take our analysis a step further with empirical results on a new benchmark with evaluation metrics specifically for measuring extrapolation performance. From this, we identify two arithmetic operations to further investigate, namely multiplication and division. For multiplication, we show how stochasticity can be applied to alleviate issues regarding falling into local minimas which cannot extrapolate. For division, we show through an extensive set of empirical results the mechanisms which can aid and hinder robustness. Factors other than the architecture are investigated including using images as the input modality, using a different loss criterion and feature scaling. In the final chapter, we draw inspiration from a human cognitive theory, the Global Workspace Theory (GWT), to develop an end-to-end architecture to combine different NALMs for compositional arithmetic.

More information

Identifiers

ORCID for Bhumika Mistry: orcid.org/0000-0003-4555-0121

ORCID for Kate Farrahi: orcid.org/0000-0001-6775-127X

ORCID for Jonathon Hare: orcid.org/0000-0003-2921-4283

Catalogue record

Export record