TY - JOUR

T1 - The Reliability of Rating via Audio- Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC)

AU - Floyd, E.

AU - Shelley, A.

AU - Crane, Rebecca

AU - Brewer, Judson

AU - Moran, P.

AU - Richler, R.

AU - Hartogensis, W.

AU - Kuyken, Willem

AU - Hecht, Frederick M.

N1 - © The Author(s) 2023.

PY - 2023

Y1 - 2023

N2 - BACKGROUND: The Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC) is an important tool for assessing teacher skill and aspects of the fidelity of mindfulness-based interventions, but prior research on and implementation of the MBI:TAC has used video recordings, which can be difficult to obtain, share for assessments, and which increase privacy concerns for participants. Audio-only recordings might be a useful alternative, but their reliability is unknown.OBJECTIVE: To assess evaluator perception of the rating process and inter-rater reliability of MBI:TAC ratings using audio-only recordings.METHODS: We prepared audio-only files from video recordings of 21 previously rated Mindfulness-Based Stress Reduction teachers. Each audio recording was rated by 3 trained MBI:TAC assessors drawn from a pool of 12 who had previously participated in rating the video recordings. Teachers were rated by evaluators who had not viewed the video recording and did not know the teacher. We then conducted semi-structured interviews with evaluators.RESULTS: On the 6 MBI:TAC domains, the intraclass correlation coefficients (ICCs) for audio recordings ranged from .53 to .69 using an average across 3 evaluators. Using a single rating resulted in lower ICCs (.27-.38). Bland-Altman plots showed audio ratings had little consistent bias compared to video recordings and agreed more closely for teachers with higher ratings. Qualitative analysis identified 3 themes: video recordings were particularly helpful when rating less skillful teachers, video recordings tended to provide a more complete picture for rating, and audio rating had some positive features.CONCLUSIONS: Inter-rater reliability of the MBI:TAC using audio-only recordings was adequate for many research and clinical purposes, and reliability is improved when using an average across several evaluators. Ratings using audio-only recordings may be more challenging when rating less experienced teachers.

AB - BACKGROUND: The Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC) is an important tool for assessing teacher skill and aspects of the fidelity of mindfulness-based interventions, but prior research on and implementation of the MBI:TAC has used video recordings, which can be difficult to obtain, share for assessments, and which increase privacy concerns for participants. Audio-only recordings might be a useful alternative, but their reliability is unknown.OBJECTIVE: To assess evaluator perception of the rating process and inter-rater reliability of MBI:TAC ratings using audio-only recordings.METHODS: We prepared audio-only files from video recordings of 21 previously rated Mindfulness-Based Stress Reduction teachers. Each audio recording was rated by 3 trained MBI:TAC assessors drawn from a pool of 12 who had previously participated in rating the video recordings. Teachers were rated by evaluators who had not viewed the video recording and did not know the teacher. We then conducted semi-structured interviews with evaluators.RESULTS: On the 6 MBI:TAC domains, the intraclass correlation coefficients (ICCs) for audio recordings ranged from .53 to .69 using an average across 3 evaluators. Using a single rating resulted in lower ICCs (.27-.38). Bland-Altman plots showed audio ratings had little consistent bias compared to video recordings and agreed more closely for teachers with higher ratings. Qualitative analysis identified 3 themes: video recordings were particularly helpful when rating less skillful teachers, video recordings tended to provide a more complete picture for rating, and audio rating had some positive features.CONCLUSIONS: Inter-rater reliability of the MBI:TAC using audio-only recordings was adequate for many research and clinical purposes, and reliability is improved when using an average across several evaluators. Ratings using audio-only recordings may be more challenging when rating less experienced teachers.

U2 - 10.1177/27536130221149966

DO - 10.1177/27536130221149966

M3 - Article

C2 - 37216036

VL - 12

JO - Global Advances in Health and Medicine

JF - Global Advances in Health and Medicine

SN - 2164-9561

ER -