Gates Foundation’s MET Study Challenged

A review by the National Education Policy Center (NEPC) of a newly released and long-awaited study on teacher evaluation strongly questions the spin that has been put on the findings.

The Measures of Effective Teaching (MET) project, funded by the Bill and Melinda Gates Foundation, released its final set of reports this month. Those reports are supposed to advise schools and districts about how to design teacher evaluations.

The MET study compared three types of teacher performance measures: Student test scores, classroom observations, and student surveys. The project concluded that the three should be given roughly equal weight in teacher evaluations.

The reviewers found that the data do not support that conclusion.

Here are some of the issues NEPC’s reviewers found with the MET study:

Samples Were Not Representative of the Teaching Force

The centerpiece of the MET study was an experiment that randomly assigned students to teachers. This experimental approach was meant to determine once and for all whether value-added (VA) scores are biased by student assignments. That is, do teachers who are assigned more successful students benefit in terms of their VA scores? But the group of teachers who participated in the MET experiment turned out not to be representative of teachers as a whole, and many participating schools failed to comply with their experimental assignments. As a result, the experiment did little to resolve the question.

No Single “Quality” Factor

Each type of measure explored in the MET study (student test scores, classroom observations, and student surveys) captures an independent dimension of teaching practice. But each measure provides only minimal information about the others. These results indicate that there is no single general teaching “quality” factor—or that if there is any such factor it accounts for only a small share of the variation in each of the measures. Rather, there are a number of distinct factors, and policymakers must choose how to weight them in designing evaluations.

MET Results Will Not Lead to More Effective Teachers

None of the three types of performance measures captures much of the variation in teachers’ impacts on alternative, conceptually demanding tests. There is little reason to believe that an evaluation system based on any of the measures considered in the MET project will do a good job of identifying teachers who are effective (or ineffective) at raising students’ performance on these more conceptually demanding assessments.

The MET review is published by the National Education Policy Center, housed at the University of Colorado Boulder School of Education.