With an increase in large scale Applied Behavior Analysis (ABA) services for children with autism, the need to define and measure quality is essential. Staff competence is key and identifying and measuring this accurately is critical. ABA service providers typically measure competence by direct observation, video analysis, and written examination. However, apart from the York Measure of Quality of Intensive Behavioural Intervention (YMQI) there is an interesting lack of direct links between defining competencies and developing assessment tools. In this study we used three measures of competencies developed from the UK ABA Autism Education Competence Framework Level 1. Along with the YMQI we assessed their construct validity by comparing the performance of two groups of tutors working in a school for children with autism (“experienced” vs. “inexperienced”) and performance of the “inexperienced” group at baseline (T1) and following one year of competence based training (T2). Results revealed that the more experienced group in both the between group and longitudinal comparisons achieved higher scores supporting the construct validity of the measures. There were few associations between the different methods of assessing competence, suggesting that no measure should be used in isolation if competence is to be comprehensively assessed.