Claims of Achievement Growth in Newark Fall Flat

Key Review Takeaway: Little evidence is provided in support of Zuckerberg-funded reforms.

Two recent reports purport to offer a comprehensive assessment of the impact on student achievement of Mark Zuckerberg’s $100 million grant to Newark schools. Yet these first attempts at such an assessment come up far short, according to a new review.

 

School District Reform in Newark: Within- and Between-School Changes in Achievement Growth, published by the National Bureau of Economic Research, and Assessing the Impact of the Newark Education Reforms: The Role of Within-School Improvement vs. Between-School Shifts in Enrollment, published by the Center for Education Policy Research, Harvard University, are both authored by Mark J. Chin, Thomas J. Kane, Whitney Kozakowski, Beth E. Schueler, and Douglas O. Staiger. Mark Weber, a doctoral candidate at Rutgers, and Bruce D. Baker, a Professor at Rutgers, reviewed the reports.

Weber and Baker found that the reports do not clearly define the treatment in question, omit important factors that are known to affect student learning and test score outcomes, are hampered by the use of crude data, and find what can, at best, be described as isolated and small effect sizes.

According to the reports, the Zuckerberg donation initiated a series of “reforms” in the Newark schools. These reforms are divided into “within-school” (personnel changes, Common Core implementation, turnaround schools, and a teacher contract featuring differentiated pay) and “between-school” (school closures, charter school expansion, and universal enrollment) components.

The reforms were supposed to spur increases in student achievement growth; however, the reports find no increase in student growth or “value-added” in math and only nominal increases in English language arts over the five-year period following the grant. The reviewers note that these small gains in English are most likely due to a change in assessments, rather than to any policies connected to the Zuckerberg donation. They also note that many districts close to Newark with similar demographics experienced similar gains in ELA relative to the rest of the state, further calling into question any causal inference that the gains had anything to do with policy changes in Newark.

In addition, there is little evidence presented as to how the reforms were actually implemented, or how they differed from other New Jersey schools, making any claim of a causal connection between the grant, the reforms, and student achievement growth suspect.

A central contention of the reports is that the majority of the small gain in English was due to the “between-school” reforms: students moving from less productive to more productive schools – specifically, to charter schools. Weber and Baker find, however, that (in addition to the above-noted problems) the reports did not account for critical differences between Newark’s district schools and charter schools. Key differences include resources, student characteristics, discipline, student attrition, staffing, and curricular narrowing.

Assuming that the “between-schools” locus is correct, the underlying change might be “charterness” or might be one or more of these related differences that have little to do with charterness. The results, therefore, are rendered inconclusive and provide no evidence in favor of the Newark-Zuckerberg reforms or the efficacy of moving students in urban districts to charter schools.

The reports repeatedly claim to be a “productivity” analysis—a type of analysis that evaluates which schools or districts get the most “bang for the buck.” The reports, however, make no attempt to account for differences in school “inputs” – the resource differences that can have a profound effect on student achievement. Because Newark charter schools enjoy significant resource advantages over district schools, omitting those advantages from the analyses greatly diminishes the value of these reports for shaping the education policies of Newark schools.

For these reasons and others, Weber and Baker conclude, the reports provide little evidence in favor of the Zuckerberg-funded reforms, particularly when considering the documented disruption around Newark’s schools that has occurred since 2010.

Find the review, by Mark Weber and Bruce B. Baker, at:
http://nepc.colorado.edu/thinktank/review-newark-reform