An investigation into weight fixing networks

Abstract

Deep neural networks require vast computational resources, with data movement between memory and processors dominating energy consumption. This thesis presents novel compression techniques that can reduce these costs by minimising the number of unique parameters in neural networks.

We developed Weight Fixing Networks (WFN), which achieve lossless compression using 56x fewer unique weights and 1.9x lower weight-space entropy than state-of-the-art quantisation approaches. However, this revealed a key limitation: treating all weights based solely on value neglects their position-specific importance.

To address this, we introduced Probabilistic Weight Fixing Networks (PWFN), employing Bayesian neural networks to learn position-specific weight uncertainties. This probabilistic framework enables more intelligent compression decisions, reducing a 5-million parameter transformer to just 296 unique values whilst outperforming state-of-the-art quantisation by 1.6% top-1 accuracy on ImageNet.

Our final contribution, Cluster-On-the-Fly PWFN, integrates clustering directly into training, streamlining the compression process. Unexpectedly, these compressed networks also provide well-calibrated uncertainty estimates, enhancing reliability alongside efficiency.

By focusing on whole-network weight reuse and targeting hardware-friendly values, this work demonstrates that neural networks can be compressed far beyond current limits whilst improving both accuracy and uncertainty quantification. These findings have significant implications for deploying efficient and trustworthy AI on resource-constrained devices.

More information

Identifiers

ORCID for Srinandan Dasmahapatra: orcid.org/0000-0002-9757-5315

ORCID for Alex Weddell: orcid.org/0000-0002-6763-5460

Catalogue record

Export record