Risk of bias in effect estimates is real but still largely ignored in environmental research

Primary and secondary research can provide valuable data on the estimation of the size of effects (effect estimates) for interventions or exposures. However, if there are any limitations in the design or conduct of the research or if there is selective reporting of the data or results, then these effect estimates may be biased. Biased estimates may lead to misinformed decisions. In health and medical sciences, there are many studies that evaluate the potential impacts of biases in effect estimates due to specific limitations of the study design or implementation, or selective reporting. In contrast, empirical research evaluating the potential impacts of biases has yet to be explored in the environmental sector. In this paper, we conducted searches for environmental research that quantitatively assessed the potential impacts of biases in effect estimates using real-world data. Previous research has identified 121 types of bias relevant to the estimation of causal effects in primary and secondary environmental research. We found 27 papers evaluating the potential impacts of 39 (out of 121) types of bias. The most studied type of bias was confounding bias (12 articles) followed by detection bias (seven articles), measurement bias (five articles), meta-analysis bias (five articles), and selection bias (three articles). The other 34 studied types of bias were evaluated in one or two papers. Based on the empirical evidence provided in these included papers, we provide implications for environmental scientists and a suggestion that authors, peer reviewers and journals editors use existing tools to identify risk of bias when planning, conducting, reporting and evaluating research. Although we do not review literature exhaustively, our findings suggest that risk of bias was detected and there are significant knowledge gaps. Future research is needed to better understand the nature and impact of potential types of bias, especially for the 82 unstudied ones.