New York City School Choice Report Mistakes Correlation for Causation

A recent Brookings Institution report asserting that school choice and competition helped improve New York City students’ test scores and graduation rates is merely “weakly supported advocacy” – not reliable research.

That’s the conclusion of a new scholarly review of the report. The review was written by Patricia Burch of the University of Southern California, Mary Stewart of the University of Wisconsin-Madison, and Jahni Smith, also of USC. The review was produced for the Think Twice think tank review project and is published today by the National Education Policy Center, housed at the University of Colorado Boulder School of Education.

Burch, Stewart, and Smith reviewed School Choice and School Performance in the New York City Public Schools—Will the Past Be Prologue?, written for Brookings by Grover (Russ) Whitehurst and Sarah Whitfield and published in October.

The Brookings report contended that school choice and competition have contributed to improved test scores and graduation rates in New York City. Mayor Michael Bloomberg implemented universal high school choice in the city’s public schools in 2004.

The reviewers point out that in contrast to a lengthy introductory section of the report that extolls the purported benefits of choice and competition, “only four pages are dedicated to discussing results, and many of those contentions are problematic.”

The Brookings report used data from several other recent reports to conclude Bloomberg’s school reform strategy has improved academic performance for district students, including low-income and disadvantaged students.

“However, these conclusions are mainly based on causal interpretations of correlational data, and the findings are presented selectively,” the reviewers write. Indeed, while the Brookings authors acknowledge the importance of not confusing correlation with causation, they nonetheless proceed to make unwarranted causal inferences as they interpret the New York City data.

Moreover, the report’s recommendations promote expanding choice and competition in accordance with the Bloomberg strategy and with Brookings’ own “Education Choice and Competition Index” – but without any logical link to the evidence presented in the report.

In producing “weakly supported advocacy… rather than research,” the reviewers conclude, the Brookings paper has little if anything to offer by way of informed guidance for policy.