TY - JOUR

T1 - The reliability of evidence review methodology in environmental science and conservation

AU - O'Leary, Bethan

AU - Kvist, Kristian

AU - Bayliss, H.R.

AU - Derroire, Geraldine

AU - Healey, John

AU - Hughes, Kathryn

AU - Kleinschroth, Fritz

AU - Sciberras, Marija

AU - Woodcock, Paul

AU - Pullin, Andrew

PY - 2016/10/1

Y1 - 2016/10/1

N2 - Given the proliferation of primary research articles, the importance of reliable environmentalevidence reviews for informing policy and management decisions is increasing. Although conductingreviews is an efficient method of synthesising the fragmented primary evidence base, reviews thatare of poor methodological reliability have the potential to misinform by not accurately reflectingthe available evidence base. To assess the current value of evidence reviews for decision-making weappraised a systematic sample of articles published in early 2015 (N=92) using the Collaboration forEnvironmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology ofpolicy-relevant evidence reviews according to elements important for objectivity, transparency andcomprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modalscore of zero (range 0-30, mean 5.8), and low scores were ubiquitous across subject areas. Ingeneral, reviews that applied meta-analytical techniques achieved higher scores than narrativesyntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing toadequately report methodology or how conclusions were drawn. However, some narrativesyntheses achieved high scores, illustrating that the reliability of reviews should be assessed on acase-by-case basis. Given the potential importance of reviews for informing management and policy,as well as research, it is vital that overall methodological reliability is improved. Although theincreasing number of systematic reviews and meta-analyses highlight that some progress is beingmade, our findings suggest little or no improvement in the last decade. To motivate progress, werecommend that an annual assessment of the methodological reliability of evidence reviews beconducted. To better serve the environmental policy and management communities we identify arequirement for independent critical appraisal of review methodology thus enabling decisionmakersto select reviews that are most likely to accurately reflect the evidence base.

AB - Given the proliferation of primary research articles, the importance of reliable environmentalevidence reviews for informing policy and management decisions is increasing. Although conductingreviews is an efficient method of synthesising the fragmented primary evidence base, reviews thatare of poor methodological reliability have the potential to misinform by not accurately reflectingthe available evidence base. To assess the current value of evidence reviews for decision-making weappraised a systematic sample of articles published in early 2015 (N=92) using the Collaboration forEnvironmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology ofpolicy-relevant evidence reviews according to elements important for objectivity, transparency andcomprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modalscore of zero (range 0-30, mean 5.8), and low scores were ubiquitous across subject areas. Ingeneral, reviews that applied meta-analytical techniques achieved higher scores than narrativesyntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing toadequately report methodology or how conclusions were drawn. However, some narrativesyntheses achieved high scores, illustrating that the reliability of reviews should be assessed on acase-by-case basis. Given the potential importance of reviews for informing management and policy,as well as research, it is vital that overall methodological reliability is improved. Although theincreasing number of systematic reviews and meta-analyses highlight that some progress is beingmade, our findings suggest little or no improvement in the last decade. To motivate progress, werecommend that an annual assessment of the methodological reliability of evidence reviews beconducted. To better serve the environmental policy and management communities we identify arequirement for independent critical appraisal of review methodology thus enabling decisionmakersto select reviews that are most likely to accurately reflect the evidence base.

U2 - 10.1016/j.envsci.2016.06.012

DO - 10.1016/j.envsci.2016.06.012

M3 - Article

VL - 64

SP - 75

EP - 82

JO - Environmental Science and Policy

JF - Environmental Science and Policy

SN - 1462-9011

ER -