Despite calls for greater use of randomized control trials (RCTs) to evaluate the impact of conservation interventions; such experimental evaluations remain extremely rare. Payments for environmental services (PES) are widely used to slow tropical deforestation but there is widespread recognition of the need for better evidence of effectiveness. A Bolivian nongovernmental organization took the unusual step of randomizing the communities where its conservation incentive program (Watershared) was offered. We explore the impact of the program on deforestation over 5 years by applying generalized additive models to Global Forest Change (GFC) data. The “intention‐to‐treat” model (where units are analyzed as randomized regardless of whether the intervention was delivered as planned) shows no effect; deforestation did not differ between the control and treatment communities. However, uptake of the intervention varied across communities so we also explored whether higher uptake might reduce deforestation. We found evidence of a small effect at high uptake but the result should be treated with caution. RCTs will not always be appropriate for evaluating conservation interventions due to ethical and practical considerations. Despite these challenges, randomization can improve causal inference and deserves more attention from those interested in improving the evidence base for conservation.