Measures of Effective Teaching

Ω


This report presents an in-depth discussion of the analytical methods and findings from the Bill & Melinda Gates Foundation's Measures of Effective Teaching (MET) project’s analysis of classroom observations. A nontechnical companion report describes implications for policymakers and practitioners. Together, these two documents on classroom observations represent the second pair of publications from the MET project.

In December 2010, the project released its initial analysis of measures of student perceptions and student achievement in Learning about Teaching: Initial Findings from the Measures of Effective Teaching Project. Two more reports are planned for mid-2012: one on the implications of assigning weights to different measures; another testing the validity of teacher effectiveness measures following random assignment of students to teachers.

There is a growing consensus that teacher evaluation in the United States is fundamentally broken. Few would argue that a system that tells 98 percent of teachers they are “satisfactory” benefits anyone—including teachers. The nation’s collective failure to invest in high-quality professional feedback to teachers is inconsistent with decades of research reporting large disparities in student learning gains in different teachers’ classrooms (even within the same schools). The quality of instruction matters. And our schools pay too little attention to it. Many states and school districts are looking to reinvent the way they do teacher evaluation and feedback, and they want better tools. With the help of nearly 3,000 teacher-volunteers, the Measures of Effective Teaching (MET) project is evaluating alternative ways to provide valid and reliable feedback to teachers for professional development and improvement.

The researchers investigated the following five instruments in this report:

__ Framework for Teaching (or FFT, developed by Charlotte Danielson of the Danielson Group),
__ Classroom Assessment Scoring System (or CLASS , developed by Robert Pianta, Karen La Paro, and Bridget Hamre at the University of Virginia),
__ Protocol for Language Arts Teaching Observations (or PLATO, developed by Pam Grossman at Stanford University),
__ Mathematical Quality of Instruction (or MQI, developed by Heather Hill of Harvard University), and
__ UTeach Teacher Observation Protocol (or UTOP, developed by Michael Marder and Candace Walkington at the University of Texas-Austin).


Findings

The analysis in this report is based on the practice of 1,333 teachers from the following districts: Charlotte- Mecklenburg, N.C.; Dallas; Denver; Hillsborough Co., Fla.; New York City; and Memphis. This is the subset of MET project volunteers who taught math or English language arts (ELA) in grades 4 through 8 and who agreed to participate in random assignment during year 2 of the project.

Five findings stood out:

1. All five observation instruments were positively associated with student achievement gains.

2. Reliably characterizing a teacher’s practice requires averaging scores over multiple observations.

3. Combining observation scores with evidence of student achievement gains and student feedback improved predictive power and reliability.

4. In contrast to teaching experience and graduate degrees, the combined measure identifies teachers with larger gains on the state tests.

Compared to teachers with fewer than three years of experience, teachers with 12 or more years of experience had students with slightly higher achievement gains in math (0.5 months higher) and slightly higher achievement gains in ELA (0.7 months higher). The difference for those with master’s degrees was also small: 1 month higher in math and actually 0.6 months lower in ELA compared to those teachers without master’s.

5. Teachers with strong performance on the combined measure also performed well on other student outcomes:

__ Their students showed larger performance gains on tests of conceptual understanding in mathematics and a literacy test requiring short-answer responses.
__ Their students reported higher levels of effort and greater enjoyment in class.

Ω