Understanding Bias in Effect Estimates and Refining a Critical Appraisal Tool for Risk of Bias Assessment in Environmental Evidence Synthesis

Abstract

Evidence syntheses that address causal questions are a key source of evidence for decision-makers, researchers, and the public. However, for evidence syntheses to be reliable, a risk-of-bias assessment is necessary. Bias can distort effect estimates derived from primary studies, potentially resulting in misleading conclusions when these studies are synthesised. Flaws in the conduct of the evidence synthesis can also lead to bias. Although risk-of-bias assessment is well-recognised in health and medical sciences, its application in environmental evidence syntheses has been relatively inconsistent and inadequate. In response to this gap, the Collaboration for Environmental Evidence (CEE) released a prototype tool in 2020 to support risk-of-bias assessment. This thesis aims to evaluate the comprehensiveness of the tool and to improve the tool’s clarity and user-friendliness through three main objectives: identifying relevant types of bias, reviewing the impacts of biases, and testing the tool’s usability in real-world contexts.
Chapter 2 identifies the types of bias relevant to the estimation of effects in environmental research and maps these against the domains of bias included in the tool. Drawing primarily on the Catalogue of Bias and key literature, 121 out of 206 recognised types of bias were identified as relevant. These were mapped against the seven domains for primary research included in the tool: (1) confounding, (2) post-intervention/selection, (3) measurement of intervention/exposure, (4) performance, (5) measurement of outcome, (6) reporting, and (7) outcome assessment, and four domains for secondary research: (1) searching, (2) screening, (3) data coding/extraction and study appraisal, and (4) data synthesis. This research provides the foundation for an understanding of how bias can occur. The conclusion is that the tool is not missing any relevant domains.
Chapter 3 explores the real-world impacts of these biases. While health and medical sciences have a well-established body of empirical evidence on the impacts of bias, such research is limited in environmental sciences. A review of 27 articles showed that only 39 of the 121 identified biases have been empirically examined. Confounding bias was the most frequently studied, followed by detection and measurement biases. The findings underscore the need for further empirical research, particularly on the 82 unstudied biases, and reinforce the importance of using available tools to identify and mitigate risk of bias.
Chapter 4 evaluates the clarity and user-friendliness of the tool based on survey feedback from authors who had used it in their evidence syntheses. With a 59% response rate, responses indicated neutral perceptions of overall clarity and user-friendliness, and positive ratings for ease of making risk of bias judgements. Thematic analysis revealed areas for improvement, including clearer terminology, enhanced instructions, and additional examples. The survey confirmed the tool’s structure was sound and did not require fundamental redesign, though refinements were necessary.
Chapter 5 details the revisions made based on the obtained feedback. The language was thoroughly revised for clarity, figures were updated to a colour-blind safe format, and explanatory materials were added. The underlying structure—comprising risk-of-bias criteria, checklist questions, and judgement algorithms—remained unchanged. This revised version is intended for submission to CEE.
Chapter 6 reflects on the evaluation processes and revisions. It identifies future research priorities, including the further definition and empirical evaluation of biases, and further evaluation of the tool such as inter-rater reliability and the time required for completing risk-of-bias assessments. It also provides three recommendations for raising the awareness of risk-of-bias: targeted user support, support for journal editors and peer reviewers, and formal education. Although limitations exist, this PhD project was a realistic and practical means of improving the tool. The revised version can now be used with greater confidence. It is concluded that all three objectives were achieved.

Date of Award	24 Aug 2026
Original language	English
Awarding Institution	Bangor University
Supervisor	James Gibbons (Supervisor), Andrew Pullin (Supervisor) & Ruth Lewis (Supervisor)