Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology


Contact
helmut.hillebrand [ at ] hifmb.de

Abstract

Field studies are essential to reliably quantify ecological responses to global change because they are exposed to realistic climate manipulations. Yet such studies are limited in replicates, resulting in less power and, therefore, potentially unreliable effect estimates. Furthermore, while manipulative field experiments are assumed to be more powerful than non-manipulative observations, it has rarely been scrutinized using extensive data. Here, using 3847 field experiments that were designed to estimate the effect of environmental stressors on ecosystems, we systematically quantified their statistical power and magnitude (Type M) and sign (Type S) errors. Our investigations focused upon the reliability of field experiments to assess the effect of stressors on both ecosystem's response magnitude and variability. When controlling for publication bias, single experiments were underpowered to detect response magnitude (median power: 18%–38% depending on effect sizes). Single experiments also had much lower power to detect response variability (6%–12% depending on effect sizes) than response magnitude. Such underpowered studies could exaggerate estimates of response magnitude by 2–3 times (Type M errors) and variability by 4–10 times. Type S errors were comparatively rare. These observations indicate that low power, coupled with publication bias, inflates the estimates of anthropogenic impacts. Importantly, we found that meta-analyses largely mitigated the issues of low power and exaggerated effect size estimates. Rather surprisingly, manipulative experiments and non-manipulative observations had very similar results in terms of their power, Type M and S errors. Therefore, the previous assumption about the superiority of manipulative experiments in terms of power is overstated. These results call for highly powered field studies to reliably inform theory building and policymaking, via more collaboration and team science, and large-scale ecosystem facilities. Future studies also require transparent reporting and open science practices to approach reproducible and reliable empirical work and evidence synthesis.



Item Type
Article
Authors
Divisions
Primary Division
Programs
Primary Topic
Publication Status
Published
Eprint ID
58029
DOI 10.1111/gcb.15972

Cite as
Yang, Y. , Hillebrand, H. , Lagisz, M. , Cleasby, I. and Nakagawa, S. (2022): Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology , Global Change Biology, 28 (3), pp. 969-989 . doi: 10.1111/gcb.15972


Download
[thumbnail of Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in gl.pdf]
Preview
PDF
Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in gl.pdf

Download (3MB) | Preview

Share
Add to AnyAdd to TwitterAdd to FacebookAdd to LinkedinAdd to PinterestAdd to Email


Citation


Actions
Edit Item Edit Item