TY - JOUR

T1 - Ignoring non‐English‐language studies may bias ecological meta‐analyses

AU - Konno, Ko

AU - Akasaka, Munemitsu

AU - Koshida, Chieko

AU - Katayama, Naoki

AU - Osada, Noriyuki

AU - Spake, Rebecca

AU - Amano, Tatsuya

PY - 2020/7/1

Y1 - 2020/7/1

N2 - Meta-analysis plays a crucial role in syntheses of quantitative evidence in ecology and biodiversity conservation. The reliability of estimates in meta-analyses strongly depends on unbiased sampling of primary studies. Although earlier studies have explored potential biases in ecological meta-analyses, biases in reported statistical results and associated study characteristics published in different languages have never been tested in environmental sciences. We address this knowledge gap by systematically searching published meta-analyses and comparing effect-size estimates between English- and Japanese-language studies included in existing meta-analyses. Of the 40 published ecological meta-analysis articles authored by those affiliated to Japanese institutions, we find that three meta-analysis articles searched for studies in the two languages and involved sufficient numbers of English- and Japanese-language studies, resulting in four eligible meta-analyses (i.e., four meta-analyses conducted in the three meta-analysis articles). In two of the four, effect sizes differ significantly between the English- and Japanese-language studies included in the meta-analyses, causing considerable changes in overall mean effect sizes and even their direction when Japanese-language studies are excluded. The observed differences in effect sizes are likely attributable to systematic differences in reported statistical results and associated study characteristics, particularly taxa and ecosystems, between English- and Japanese-language studies. Despite being based on a small sample size, our findings suggest that ignoring non-English-language studies may bias outcomes of ecological meta-analyses, due to systematic differences in study characteristics and effect-size estimates between English- and non-English languages. We provide a list of actions that meta-analysts could take in the future to reduce the risk of language bias.

AB - Meta-analysis plays a crucial role in syntheses of quantitative evidence in ecology and biodiversity conservation. The reliability of estimates in meta-analyses strongly depends on unbiased sampling of primary studies. Although earlier studies have explored potential biases in ecological meta-analyses, biases in reported statistical results and associated study characteristics published in different languages have never been tested in environmental sciences. We address this knowledge gap by systematically searching published meta-analyses and comparing effect-size estimates between English- and Japanese-language studies included in existing meta-analyses. Of the 40 published ecological meta-analysis articles authored by those affiliated to Japanese institutions, we find that three meta-analysis articles searched for studies in the two languages and involved sufficient numbers of English- and Japanese-language studies, resulting in four eligible meta-analyses (i.e., four meta-analyses conducted in the three meta-analysis articles). In two of the four, effect sizes differ significantly between the English- and Japanese-language studies included in the meta-analyses, causing considerable changes in overall mean effect sizes and even their direction when Japanese-language studies are excluded. The observed differences in effect sizes are likely attributable to systematic differences in reported statistical results and associated study characteristics, particularly taxa and ecosystems, between English- and Japanese-language studies. Despite being based on a small sample size, our findings suggest that ignoring non-English-language studies may bias outcomes of ecological meta-analyses, due to systematic differences in study characteristics and effect-size estimates between English- and non-English languages. We provide a list of actions that meta-analysts could take in the future to reduce the risk of language bias.

KW - Ecology

KW - Ecology, Evolution, Behavior and Systematics

KW - Nature and Landscape Conservation

U2 - 10.1002/ece3.6368

DO - 10.1002/ece3.6368

M3 - Article

VL - 10

SP - 6373

EP - 6384

JO - Ecology and Evolution

JF - Ecology and Evolution

SN - 2045-7758

IS - 13

ER -