State tests used for VAM are not meaningfully associated with either the content or quality of instruction

Recent years have seen the convergence of two major policy streams in U.S. K–12 education: standards/accountability and teacher quality reforms. Work in these areas has led to the creation of multiple measures of teacher quality, including measures of their instructional alignment to standards/assessments, observational and student survey measures of pedagogical quality, and measures of teachers’ contributions to student test scores. 

This article is the first to explore the extent to which teachers’ instructional alignment is associated with their contributions to student learning and their effectiveness on new composite evaluation measures using data from the Bill & Melinda Gates Foundation’s Measures of Effective Teaching study.

The study concludes that:

"there are very weak associations of content alignment with student achievement gains and no associations with the composite measure of effective teaching.

 

One interpretation of this conclusion is instructional alignment and pedagogical quality are not as important as standards-based reform theory suggests for affecting student learning.

A second interpretation is that instructional alignment and pedagogical quality are as important as previously thought, but that the methods of review do not capture the important elements of instruction. 

Given the volume of literature that links students’ opportunity to learn (OTL) to achievement and the quality of the instructional measures used in this study, either interpretation would be surprising and troubling. For one, these interpretations would suggest that instructional improvement is not likely to be a fruitful path for influencing what and how much students learn.

A third interpretation is that the tests used for calculating VAM are not particularly able to detect differences in the content or quality of classroom instruction. That is, rather than the instructional measures being the cause of our low correlations, it is the tests that are the cause. Both state standardized tests and the alternate assessments used in the MET study are quite distal to the day-to-day instruction occurring in classrooms (Ruiz-Primo et al., 2002). Empirical and conceptual work illustrates that these kind of assessments tend to be, at best, weakly sensitive to carefully measured indicators of instructional content or quality…

At a minimum, these results suggest it may be fruitless for teachers to use state test VAMs to inform adjustments to their instruction. Furthermore, this interpretation raises the question— If VAMs are not meaningfully associated with either the content or quality of instruction, what are they measuring?"