Deciphering ecology from statistical artefacts: Competing influence of sample size, prevalence and habitat specialization on species distribution models and how small evaluation datasets can inflate metrics of performance
Research output: Contribution to journal › Article › peer-review
Standard Standard
In: Diversity and Distributions, Vol. 26, No. 3, 09.01.2020, p. 315-328.
Research output: Contribution to journal › Article › peer-review
HarvardHarvard
APA
CBE
MLA
VancouverVancouver
Author
RIS
TY - JOUR
T1 - Deciphering ecology from statistical artefacts: Competing influence of sample size, prevalence and habitat specialization on species distribution models and how small evaluation datasets can inflate metrics of performance
AU - Hallman, Tyler A
AU - Robinson, W Douglas
PY - 2020/1/9
Y1 - 2020/1/9
N2 - AimSample size and species characteristics, including prevalence and habitat specialization, can influence the predictive performance of species distribution models (SDMs). There is little agreement, however, on which metric of model performance to use. Here, we directly compare AUC and partial ROC as metrics of SDM performance through analyses on the effects of species traits and sample size on SDM performance.LocationThree counties dominated by agricultural lands and coniferous forest in Oregon's Willamette Valley and Coast Range ecoregions.MethodsWe systematically reduced a large avian point count dataset to alter sample sizes of 22 species of songbird. We used boosted regression trees to run SDMs for each species, quantified habitat specialization, and used mixed effects models to compare the influence of sample size, prevalence, and habitat specialization on SDM performance, calculated as AUC and partial ROC, across species. We calculated AUC and partial ROC with subset and independent evaluation data separately to more comprehensively investigate differences in metrics.ResultsWe found a positive quadratic effect of sample size and a strongly positive effect of habitat specialization on both metrics of model performance. We found a weak effect of prevalence on partial ROC and no effect in AUC. Contrary to expectations, when evaluated with a subset evaluation data, partial ROC was consistently highest in models with the smallest sample sizes. These small sample size models had correspondingly small sample sizes in subset evaluation datasets. Partial ROC evaluated with independent data and AUC evaluated with subset or independent data showed the expected positive correlation between sample size and model performance.Main ConclusionsWe found that small evaluation datasets can artificially inflate partial ROC. With literature recommended minimum SDM sample sizes as low as three, attention must be given to the effects of correspondingly low sample sizes in evaluation datasets.
AB - AimSample size and species characteristics, including prevalence and habitat specialization, can influence the predictive performance of species distribution models (SDMs). There is little agreement, however, on which metric of model performance to use. Here, we directly compare AUC and partial ROC as metrics of SDM performance through analyses on the effects of species traits and sample size on SDM performance.LocationThree counties dominated by agricultural lands and coniferous forest in Oregon's Willamette Valley and Coast Range ecoregions.MethodsWe systematically reduced a large avian point count dataset to alter sample sizes of 22 species of songbird. We used boosted regression trees to run SDMs for each species, quantified habitat specialization, and used mixed effects models to compare the influence of sample size, prevalence, and habitat specialization on SDM performance, calculated as AUC and partial ROC, across species. We calculated AUC and partial ROC with subset and independent evaluation data separately to more comprehensively investigate differences in metrics.ResultsWe found a positive quadratic effect of sample size and a strongly positive effect of habitat specialization on both metrics of model performance. We found a weak effect of prevalence on partial ROC and no effect in AUC. Contrary to expectations, when evaluated with a subset evaluation data, partial ROC was consistently highest in models with the smallest sample sizes. These small sample size models had correspondingly small sample sizes in subset evaluation datasets. Partial ROC evaluated with independent data and AUC evaluated with subset or independent data showed the expected positive correlation between sample size and model performance.Main ConclusionsWe found that small evaluation datasets can artificially inflate partial ROC. With literature recommended minimum SDM sample sizes as low as three, attention must be given to the effects of correspondingly low sample sizes in evaluation datasets.
U2 - 10.1111/ddi.13030
DO - 10.1111/ddi.13030
M3 - Article
VL - 26
SP - 315
EP - 328
JO - Diversity and Distributions
JF - Diversity and Distributions
SN - 1472-4642
IS - 3
ER -