New York City School-Based Financial Incentives Program Did Not Improve Student Achievement or Affect Reported Teaching Practices

Ω

A New York City program designed to improve student performance through school-based financial incentives for teachers did not improve student achievement, most likely because it did not change teacher behavior and the conditions needed to motivate staff were not achieved, according to a RAND Corporation study.

From 2007 to 2010, nearly 200 high-needs New York City public schools participated in the Schoolwide Performance Bonus Program. The study, commissioned by the New York City Department of Education and the United Federation of Teachers and funded by the New York City Fund for Public Schools and National Center on Performance Initiatives, is the most comprehensive study on the city's performance pay program.

Implemented for the first time in the 2007–2008 school year, this three-year program provided financial rewards based on school-level performance to educators in high-needs elementary, middle, and high schools. Schools enrolled in the program could choose to opt out.

Using independent analysis of test scores, interviews with school administrators, teachers, and other personnel, and teacher and school staff surveys, researchers say the study provides critical insight into the program's design and its implementation.

"Bonuses alone have not proven to be the answer to bettering student achievement," said Julie Marsh, the study's lead author. "Educators said bonuses are desirable, but they also said they did not change how they perform their job because of bonuses. Some didn't understand how the program worked, while others did not perceive the bonus as having tremendous value. Still others felt the bonus criteria relied too heavily on test scores. We believe these factors may have actually weakened the motivational effects of the bonus program."

The New York City Department of Education and the United Federation of Teachers, an affiliate of the American Federation of Teachers, jointly implemented the Schoolwide Performance Bonus Program for the first time during the 2007-2008 school year. Using a random sample of the city's high-needs public schools, the program lasted for three academic years, with the goal of improving student performance through school-based financial incentives paid to teachers.

Researchers from RAND Education and the National Center on Performance Initiatives at Vanderbilt University examined student test scores and administrative data; conducted teacher, school staff, and administrator surveys; and interviewed school administrators, staff members, program sponsors, and union and district officials.

The researchers found that the program did not improve student achievement or affect teachers' reported behaviors or attitudes, perhaps in part because conditions needed to motivate staff were not achieved and because of the high level of accountability already present for participating and non-participating schools.

Among the key findings:

* Overall, the program had no positive effects on student achievement at any grade level. Researcher analysis of student achievement on the state's accountability tests found no positive effects overall for students attending elementary, middle or K-8 schools in years one through three, and for high school students during the first two years of the program.
* The program did not lead to improvements on elementary, middle and high school progress report scores. The study found no statistically significant differences between scores of Schoolwide Performance Bonus Program treatment and control schools and between schools that participated in Schoolwide Performance Bonus Program each year (regardless of random assignment) and other eligible schools.
* Researchers found no differences between the reported teaching practices, effort and attitudes of teachers in treatment schools and those of the control group.
* Several key conditions that theory suggests are necessary for performance-based incentive programs to change behaviors (e.g., understanding, buy-in for the bonus criteria, perceived value of bonus, perceived fairness) did not take root in all schools.
* Other accountability incentives—such as receiving a high progress report grade or achieving adequate yearly progress targets—and intrinsic motivation were deemed by many teachers as more salient than financial rewards.

Researchers also found that a majority of the schools disseminated the bonuses equally among staff, despite program guidelines granting school committees the flexibility to distribute the bonus shares as they deemed fit.

"Other research and theory suggests that for bonus programs to be effective in improving student performance, there must be a high level of understanding of the program and bonus criteria, educators must have 'buy-in,' and they need to view bonuses as large enough to motivate extra effort, " said Marsh, adjunct researcher at RAND and visiting associate professor at the University of Southern California. "These characteristics were lacking in many schools participating in the New York City program, and were a key reason why some educators said the program did not influence them to change their behavior."

The research was conducted by RAND Education, a unit of the RAND Corporation, and National Center on Performance Initiatives partners at Vanderbilt University. Funding to carry out the work was provided by the New York City Fund for Public Schools and National Center on Performance Initiatives.