Chaotic methods for the strong scalability of CFD

Abstract

Supercomputing power has been doubling approximately every 14 months for at least three decades, increasing the capabilities of scientific modelling at a similar rate. The first machines capable of one ExaFLOP (1018 floating-point operations per second) are expected by 2020. However, architectural changes required to reach ‘exascale’ are significant, with energy efficiency constraints leading to a huge growth in parallelization. A new era of computing has arrived, dubbed the ‘many-core’ era, in which the number of computing cores is increasing faster than CFD simulation sizes – prompting the research question for this thesis:

‘What limits the strong scalability of CFD and its ability to handle many-core architectures? What can be done to improve the CFD algorithms in this respect?’

A number of scalability investigations have been performed from 1 through to 2048 cores, using a semi-implicit, finite-volume CFD code: ReFRESCO; and the University of Southampton supercomputer: Iridis4. The main bottleneck to strong scalability is shown to be the linear equation-system solvers, occupying up to 95% of total wall-time on 2048 cores – where the poor scalability arises from synchronous, global, inter-process communications. Experiments have been performed with alternative, state-of-the-art linear solvers and preconditioners, without significant improvements, which motivates novel research into scalable linear solvers for CFD.

The theory of ‘chaotic relaxation’ has been used to create a completely asynchronous Jacobi-like ‘chaotic solver’, showing almost perfect scalability, and performance far greater than their synchronous counterparts. However, these solvers lack the absolute numerical power to compete with existing solvers, especially as the resolution of the simulations increases. Following this, chaotic relaxation theory has been used to create a novel ‘chaotic-cycle’ multigrid solver, combining aspects of the chaotic solver and classical multigrid methods.

Both of the solvers have been verified and tested using canonical test cases and practical CFD simulations. On 2048 cores, the chaotic-cycle multigrid solver performs up to 7.7× faster than a typical Krylov Subspace solver and 13.3× faster than classical V-cycle multigrid. With improvements to the implementation of coarse-grid communications and desynchronized residual computations, it is likely that the chaotic-cycle multigrid method will continue scaling to many thousands of cores, thus removing the main bottleneck to the strong-scalability of CFD.

The novel chaotic solver and chaotic-cycle multigrid methods have been implemented as an open-source library, Chaos. It is hoped that work on these scalable solvers can be continued and applied to other disciplines.

More information

Identifiers

ORCID for Stephen Turnock: orcid.org/0000-0001-6288-0400

Catalogue record

Export record