TY - JOUR

T1 - Same data, different analysts

T2 - variation in effect sizes due to analytical decisions in ecology and evolutionary biology

AU - Gould, Elliot

AU - Fraser, Hannah S

AU - Parker, Timothy H

AU - Nakagawa, Shinichi

AU - Griffith, Simon C

AU - Vesk, Peter A

AU - Fidler, Fiona

AU - Hamilton, Daniel G

AU - Abbey-Lee, Robin N

AU - Abbott, Jessica K

AU - Aguirre, Luis A

AU - Alcaraz, Carles

AU - Aloni, Irith

AU - Altschul, Drew

AU - Arekar, Kunal

AU - Atkins, Jeff W

AU - Atkinson, Joe

AU - Baker, Christopher M

AU - Barrett, Meghan

AU - Bell, Kristian

AU - Bello, Suleiman Kehinde

AU - Beltrán, Iván

AU - Berauer, Bernd J

AU - Bertram, Michael Grant

AU - Billman, Peter D

AU - Blake, Charlie K

AU - Blake, Shannon

AU - Bliard, Louis

AU - Bonisoli-Alquati, Andrea

AU - Bonnet, Timothée

AU - Bordes, Camille Nina Marion

AU - Bose, Aneesh P H

AU - Botterill-James, Thomas

AU - Boyd, Melissa Anna

AU - Boyle, Sarah A

AU - Bradfer-Lawrence, Tom

AU - Bradham, Jennifer

AU - Brand, Jack A

AU - Brengdahl, Martin I

AU - Bulla, Martin

AU - Bussière, Luc

AU - Camerlenghi, Ettore

AU - Campbell, Sara E

AU - Campos, Leonardo L F

AU - Caravaggi, Anthony

AU - Cardoso, Pedro

AU - Carroll, Charles J W

AU - Catanach, Therese A

AU - Chen, Xuan

AU - Chik, Heung Ying Janet

AU - Choy, Emily Sarah

AU - Christie, Alec Philip

AU - Chuang, Angela

AU - Chunco, Amanda J

AU - Clark, Bethany L

AU - Contina, Andrea

AU - Covernton, Garth A

AU - Cox, Murray P

AU - Cressman, Kimberly A

AU - Crotti, Marco

AU - Crouch, Connor Davidson

AU - D'Amelio, Pietro B

AU - de Sousa, Alexandra Allison

AU - Döbert, Timm Fabian

AU - Dobler, Ralph

AU - Dobson, Adam J

AU - Doherty, Tim S

AU - Drobniak, Szymon Marian

AU - Duffy, Alexandra Grace

AU - Duncan, Alison B

AU - Dunn, Robert P

AU - Dunning, Jamie

AU - Dutta, Trishna

AU - Eberhart-Hertel, Luke

AU - Elmore, Jared Alan

AU - Elsherif, Mahmoud Medhat

AU - English, Holly M

AU - Ensminger, David C

AU - Ernst, Ulrich Rainer

AU - Ferguson, Stephen M

AU - Fernandez-Juricic, Esteban

AU - Ferreira-Arruda, Thalita

AU - Fieberg, John

AU - Finch, Elizabeth A

AU - Fiorenza, Evan A

AU - Fisher, David N

AU - Fontaine, Amélie

AU - Forstmeier, Wolfgang

AU - Fourcade, Yoan

AU - Frank, Graham S

AU - Freund, Cathryn A

AU - Fuentes-Lillo, Eduardo

AU - Gandy, Sara L

AU - Gannon, Dustin G

AU - García-Cervigón, Ana I

AU - Garretson, Alexis C

AU - Ge, Xuezhen

AU - Geary, William L

AU - Géron, Charly

AU - Gilles, Marc

AU - Girndt, Antje

AU - Gliksman, Daniel

AU - Goldspiel, Harrison B

AU - Gomes, Dylan G E

AU - Good, Megan Kate

AU - Goslee, Sarah C

AU - Gosnell, J Stephen

AU - Grames, Eliza M

AU - Gratton, Paolo

AU - Grebe, Nicholas M

AU - Greenler, Skye M

AU - Griffioen, Maaike

AU - Griffith, Daniel M

AU - Griffith, Frances J

AU - Grossman, Jake J

AU - Güncan, Ali

AU - Haesen, Stef

AU - Hagan, James G

AU - Hager, Heather A

AU - Harris, Jonathan Philo

AU - Harrison, Natasha Dean

AU - Hasnain, Sarah Syedia

AU - Havird, Justin Chase

AU - Heaton, Andrew J

AU - Herrera-Chaustre, María Laura

AU - Howard, Tanner J

AU - Hsu, Bin-Yan

AU - Iannarilli, Fabiola

AU - Iranzo, Esperanza C

AU - Iverson, Erik N K

AU - Jimoh, Saheed Olaide

AU - Johnson, Douglas H

AU - Johnsson, Martin

AU - Jorna, Jesse

AU - Jucker, Tommaso

AU - Jung, Martin

AU - Kačergytė, Ineta

AU - Kaltz, Oliver

AU - Ke, Alison

AU - Kelly, Clint D

AU - Keogan, Katharine

AU - Keppeler, Friedrich Wolfgang

AU - Killion, Alexander K

AU - Kim, Dongmin

AU - Kochan, David P

AU - Korsten, Peter

AU - Kothari, Shan

AU - Kuppler, Jonas

AU - Kusch, Jillian M

AU - Lagisz, Malgorzata

AU - Lalla, Kristen Marianne

AU - Larkin, Daniel J

AU - Larson, Courtney L

AU - Lauck, Katherine S

AU - Lauterbur, M Elise

AU - Law, Alan

AU - Léandri-Breton, Don-Jean

AU - Lembrechts, Jonas J

AU - L'Herpiniere, Kiara

AU - Lievens, Eva J P

AU - de Lima, Daniela Oliveira

AU - Lindsay, Shane

AU - Luquet, Martin

AU - MacLeod, Ross

AU - Macphie, Kirsty H

AU - Magellan, Kit

AU - Mair, Magdalena M

AU - Malm, Lisa E

AU - Mammola, Stefano

AU - Mandeville, Caitlin P

AU - Manhart, Michael

AU - Manrique-Garzon, Laura Milena

AU - Mäntylä, Elina

AU - Marchand, Philippe

AU - Marshall, Benjamin Michael

AU - Martin, Charles A

AU - Martin, Dominic Andreas

AU - Martin, Jake Mitchell

AU - Martinig, April Robin

AU - McCallum, Erin S

AU - McCauley, Mark

AU - McNew, Sabrina M

AU - Meiners, Scott J

AU - Merkling, Thomas

AU - Michelangeli, Marcus

AU - Moiron, Maria

AU - Moreira, Bruno

AU - Mortensen, Jennifer

AU - Mos, Benjamin

AU - Muraina, Taofeek Olatunbosun

AU - Murphy, Penelope Wrenn

AU - Nelli, Luca

AU - Niemelä, Petri

AU - Nightingale, Josh

AU - Nilsonne, Gustav

AU - Nolazco, Sergio

AU - Nooten, Sabine S

AU - Novotny, Jessie Lanterman

AU - Olin, Agnes Birgitta

AU - Organ, Chris L

AU - Ostevik, Kate L

AU - Palacio, Facundo Xavier

AU - Paquet, Matthieu

AU - Parker, Darren James

AU - Pascall, David J

AU - Pasquarella, Valerie J

AU - Paterson, John Harold

AU - Payo-Payo, Ana

AU - Pedersen, Karen Marie

AU - Perez, Grégoire

AU - Perry, Kayla I

AU - Pottier, Patrice

AU - Proulx, Michael J

AU - Proulx, Raphaël

AU - Pruett, Jessica L

AU - Ramananjato, Veronarindra

AU - Randimbiarison, Finaritra Tolotra

AU - Razafindratsima, Onja H

AU - Rennison, Diana J

AU - Riva, Federico

AU - Riyahi, Sepand

AU - Roast, Michael James

AU - Rocha, Felipe Pereira

AU - Roche, Dominique G

AU - Román-Palacios, Cristian

AU - Rosenberg, Michael S

AU - Ross, Jessica

AU - Rowland, Freya E

AU - Rugemalila, Deusdedith

AU - Russell, Avery L

AU - Ruuskanen, Suvi

AU - Saccone, Patrick

AU - Sadeh, Asaf

AU - Salazar, Stephen M

AU - Sales, Kris

AU - Salmón, Pablo

AU - Sánchez-Tójar, Alfredo

AU - Santos, Leticia Pereira

AU - Santostefano, Francesca

AU - Schilling, Hayden T

AU - Schmidt, Marcus

AU - Schmoll, Tim

AU - Schneider, Adam C

AU - Schrock, Allie E

AU - Schroeder, Julia

AU - Schtickzelle, Nicolas

AU - Schultz, Nick L

AU - Scott, Drew A

AU - Scroggie, Michael Peter

AU - Shapiro, Julie Teresa

AU - Sharma, Nitika

AU - Shearer, Caroline L

AU - Simón, Diego

AU - Sitvarin, Michael I

AU - Skupien, Fabrício Luiz

AU - Slinn, Heather Lea

AU - Smith, Grania Polly

AU - Smith, Jeremy A

AU - Sollmann, Rahel

AU - Whitney, Kaitlin Stack

AU - Still, Shannon Michael

AU - Stuber, Erica F

AU - Sutton, Guy F

AU - Swallow, Ben

AU - Taff, Conor Claverie

AU - Takola, Elina

AU - Tanentzap, Andrew J

AU - Tarjuelo, Rocío

AU - Telford, Richard J

AU - Thawley, Christopher J

AU - Thierry, Hugo

AU - Thomson, Jacqueline

AU - Tidau, Svenja

AU - Tompkins, Emily M

AU - Tortorelli, Claire Marie

AU - Trlica, Andrew

AU - Turnell, Biz R

AU - Urban, Lara

AU - Van de Vondel, Stijn

AU - van der Wal, Jessica Eva Megan

AU - Van Eeckhoven, Jens

AU - van Oordt, Francis

AU - Vanderwel, K Michelle

AU - Vanderwel, Mark C

AU - Vanderwolf, Karen J

AU - Vélez, Juliana

AU - Vergara-Florez, Diana Carolina

AU - Verrelli, Brian C

AU - Vieira, Marcus Vinícius

AU - Villamil, Nora

AU - Vitali, Valerio

AU - Vollering, Julien

AU - Walker, Jeffrey

AU - Walker, Xanthe J

AU - Walter, Jonathan A

AU - Waryszak, Pawel

AU - Weaver, Ryan J

AU - Wedegärtner, Ronja E M

AU - Weller, Daniel L

AU - Whelan, Shannon

AU - White, Rachel Louise

AU - Wolfson, David William

AU - Wood, Andrew

AU - Yanco, Scott W

AU - Yen, Jian D L

AU - Youngflesh, Casey

AU - Zilio, Giacomo

AU - Zimmer, Cédric

AU - Zimmerman, Gregory Mark

AU - Zitomer, Rachel A

N1 - © 2025. The Author(s).

PY - 2025/2/6

Y1 - 2025/2/6

N2 - Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

AB - Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

KW - Ecology/methods

KW - Biological Evolution

KW - Animals

KW - Passeriformes/physiology

KW - Eucalyptus/growth & development

U2 - 10.1186/s12915-024-02101-x

DO - 10.1186/s12915-024-02101-x

M3 - Article

C2 - 39915771

VL - 23

JO - BMC Biology

JF - BMC Biology

SN - 1741-7007

IS - 1

M1 - 35

ER -