TY - JOUR

T1 - Examining heterogeneity of education intervention effects using quantile mixed models: a re-analysis of a cluster-randomized controlled trial of a fluency-based mathematics intervention

AU - Thompson, Paul

AU - Owen, Kaydee

AU - Hastings, Richard

PY - 2024/2

Y1 - 2024/2

N2 - Traditionally, cluster randomized controlled trials are analyzed with the average intervention effect of interest. However, in populations that contain higher degrees of heterogeneity or variation may differ across different values of a covariate, which may not be optimal. Within education and social science contexts, exploring the variation in magnitude of treatment effect at different points in the population can indicate where the intervention is most effective rather than assuming an average effect.Data from [Owen, K.L., et al., 2021. Implementation support improves outcomes of a fluency-based mathematics strategy: A cluster-randomized controlled trial. Journal of research on educational effectiveness, 14 (3), 523–542.] were reanalyzed using three modeling approaches: conditional mean-modeling reporting the average treatment effect using linear mixed models, and two quantile regression-based methods. Quantile regressions report the quantile treatment effects at different percentiles: 10th, 25th, 50th, 75th and 90th. For the Quantile approaches, a significant intervention effect in the median to upper quantiles was found and linear quantile mixed model showed improved fit over the other approaches.An improved picture of intervention effects may be apparent using quantile regression methods when analyzing cluster randomized trials that have heterogeneous error variance. In particular, the linear quantile mixed model shows improved model fit allowing a multilevel framework to include random effects. There is considerable scope to extend this framework to incorporate more complex RCT designs.

AB - Traditionally, cluster randomized controlled trials are analyzed with the average intervention effect of interest. However, in populations that contain higher degrees of heterogeneity or variation may differ across different values of a covariate, which may not be optimal. Within education and social science contexts, exploring the variation in magnitude of treatment effect at different points in the population can indicate where the intervention is most effective rather than assuming an average effect.Data from [Owen, K.L., et al., 2021. Implementation support improves outcomes of a fluency-based mathematics strategy: A cluster-randomized controlled trial. Journal of research on educational effectiveness, 14 (3), 523–542.] were reanalyzed using three modeling approaches: conditional mean-modeling reporting the average treatment effect using linear mixed models, and two quantile regression-based methods. Quantile regressions report the quantile treatment effects at different percentiles: 10th, 25th, 50th, 75th and 90th. For the Quantile approaches, a significant intervention effect in the median to upper quantiles was found and linear quantile mixed model showed improved fit over the other approaches.An improved picture of intervention effects may be apparent using quantile regression methods when analyzing cluster randomized trials that have heterogeneous error variance. In particular, the linear quantile mixed model shows improved model fit allowing a multilevel framework to include random effects. There is considerable scope to extend this framework to incorporate more complex RCT designs.

KW - RCT

KW - Quantile Regression

KW - Linear Quantile Mixed Models

KW - General Linear Mixed Models

KW - Fluency-based Mathematics

U2 - 10.1080/1743727X.2023.2215699

DO - 10.1080/1743727X.2023.2215699

M3 - Article

VL - 47

SP - 49

EP - 64

JO - International Journal of Research & Method in Education

JF - International Journal of Research & Method in Education

IS - 1

M1 - 2215699

ER -