A Validation Study of the Mindfulness-Based Interventions Teaching Assessment Criteria for Assessing Mindfulness-Based Intervention Teacher Skill: Inter-Rater Reliability and Predictive Validity

Background Prior data suggests the Mindfulness-Based Interventions: (MBI) Teaching Assessment Criteria (MBI:TAC) has good inter-rater reliability, but many raters knew teacher experience level. Objective We sought to further evaluate the MBI-TAC’s inter-rater reliability and obtain preliminary data on predictive validity. Methods We videorecorded 21 MBSR teachers from academic and community settings. We trained 19 experienced MBI teachers in using the MBI:TAC. MBSR teachers were rated by three assessors; teachers and their assessors did not know one another. To assess predictive validity, MBSR students in courses taught by 18 of the MBSR teachers were invited to complete PROMIS-29 measures before the MBSR course, at the end of the course (month 2), and month 4. Results Intraclass correlation coefficients (ICCs) representing a single rater ranged from 0.33 to 0.56 on the 6 MBI:TAC domains. Using an average of two raters, ICC estimates ranged from 0.48 to 0.71 and ICCs generalizing to an average of three raters ranged from 0.6 to 0.8. Among n = 152 participating MBSR students, we found improvements from baseline to 2 months and 4 months in PROMIS measures of Anxiety, Depression, Fatigue, Sleep, and Social Role function (range in improvement 2.3 to 6.3, P < 0.0001 for all comparisons except Social Role at 2 months, P = 0.007). Higher MBI:TAC ratings were associated with greater improvements in anxiety among MBSR students from baseline to 2 months, with a −0.31 lower participant anxiety score per 1 unit increase in MBI:TAC composite teaching rating (95% CI −0.58, −0.05, P = 0.019), but we did not find statistically significant relationships with improvements in other PROMIS-29 domains. Conclusions ICCs indicated good reliability using an average of three ratings, but inter-rater reliability was only fair using a single rater. We found initial validation that higher MBI:TAC ratings predicted greater improvements in anxiety symptoms in MBSR participants.