We got the power: clarifying the role of statistical power and statistical significance in study design, inference and decision making in energy demand reduction studies

Abstract

In this paper we address ongoing confusion over the meaning of statistical significance and statistical power in energy efficiency and energy demand reduction intervention studies. We discuss the role of these concepts in designing studies, in deciding what can be inferred from the results and thus what course of subsequent action to take. We do this using a worked example of a study of Heat Pump demand response in New Zealand to show how to appropriately size experimental and observational studies, the consequences this has for subsequent data analysis and the decisions that can then be taken. The paper then provides two sets of recommendations. The first focuses on the uncontroversial but seemingly ignorable issue of statistical power analysis and sample design, something regularly omitted in the energy studies literature. The second focuses on how to report energy demand reduction study or trial results, make inferences and take commercial or policy-oriented decisions in a contextually appropriate way. The paper therefore offers guidance to researchers tasked with designing and assessing such studies; project managers who need to understand what can count as evidence, for what purpose and in what context and decision makers who need to make defensible commercial or policy decisions based on that evidence. The paper therefore helps all of them to distinguish the search for statistical significance from the requirement for actionable evidence and so avoid throwing the substantive baby out with the p-value bathwater.

More information

Identifiers

ORCID for Ben Anderson: orcid.org/0000-0003-2092-4406

ORCID for Tom Rushby: orcid.org/0000-0002-3686-5140

ORCID for Abubakr Bahaj: orcid.org/0000-0002-0043-6045

ORCID for Patrick James: orcid.org/0000-0002-2694-7054

Catalogue record

Export record