The reliability of evidence review methodology in environmental science and conservation
Research output: Contribution to journal › Article › peer-review
Electronic versions
Documents
- 2016 The reliability of evidence review methodology in environmental science and conservation
Accepted author manuscript, 752 KB, PDF document
DOI
Given the proliferation of primary research articles, the importance of reliable environmental
evidence reviews for informing policy and management decisions is increasing. Although conducting
reviews is an efficient method of synthesising the fragmented primary evidence base, reviews that
are of poor methodological reliability have the potential to misinform by not accurately reflecting
the available evidence base. To assess the current value of evidence reviews for decision-making we
appraised a systematic sample of articles published in early 2015 (N=92) using the Collaboration for
Environmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology of
policy-relevant evidence reviews according to elements important for objectivity, transparency and
comprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modal
score of zero (range 0-30, mean 5.8), and low scores were ubiquitous across subject areas. In
general, reviews that applied meta-analytical techniques achieved higher scores than narrative
syntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing to
adequately report methodology or how conclusions were drawn. However, some narrative
syntheses achieved high scores, illustrating that the reliability of reviews should be assessed on a
case-by-case basis. Given the potential importance of reviews for informing management and policy,
as well as research, it is vital that overall methodological reliability is improved. Although the
increasing number of systematic reviews and meta-analyses highlight that some progress is being
made, our findings suggest little or no improvement in the last decade. To motivate progress, we
recommend that an annual assessment of the methodological reliability of evidence reviews be
conducted. To better serve the environmental policy and management communities we identify a
requirement for independent critical appraisal of review methodology thus enabling decisionmakers
to select reviews that are most likely to accurately reflect the evidence base.
evidence reviews for informing policy and management decisions is increasing. Although conducting
reviews is an efficient method of synthesising the fragmented primary evidence base, reviews that
are of poor methodological reliability have the potential to misinform by not accurately reflecting
the available evidence base. To assess the current value of evidence reviews for decision-making we
appraised a systematic sample of articles published in early 2015 (N=92) using the Collaboration for
Environmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology of
policy-relevant evidence reviews according to elements important for objectivity, transparency and
comprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modal
score of zero (range 0-30, mean 5.8), and low scores were ubiquitous across subject areas. In
general, reviews that applied meta-analytical techniques achieved higher scores than narrative
syntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing to
adequately report methodology or how conclusions were drawn. However, some narrative
syntheses achieved high scores, illustrating that the reliability of reviews should be assessed on a
case-by-case basis. Given the potential importance of reviews for informing management and policy,
as well as research, it is vital that overall methodological reliability is improved. Although the
increasing number of systematic reviews and meta-analyses highlight that some progress is being
made, our findings suggest little or no improvement in the last decade. To motivate progress, we
recommend that an annual assessment of the methodological reliability of evidence reviews be
conducted. To better serve the environmental policy and management communities we identify a
requirement for independent critical appraisal of review methodology thus enabling decisionmakers
to select reviews that are most likely to accurately reflect the evidence base.
Original language | English |
---|---|
Pages (from-to) | 75-82 |
Journal | Environmental Science and Policy |
Volume | 64 |
Early online date | 1 Jul 2016 |
DOIs | |
Publication status | Published - 1 Oct 2016 |
Total downloads
No data available