New Orleans Charter School Study Debunked

The Center for Research on Education Outcomes (CREDO) at Stanford University recently released a follow-up report to its 2013 national study on charter school performance.

The new study is focused on Louisiana and particularly on the effects in New Orleans. The report states, “We observed positive impacts from attending a [New Orleans] charter for students in poverty (unlike in the overall state), for Black and Hispanic students, special education students, and students who repeat a grade.” Louisiana Governor Bobby Jindal called the study “proof of the success of charter schools in Louisiana.”

Such claims overstate the findings.

The study covers a period that begins right after Hurricane Katrina. New Orleans experienced immediate and dramatic shifts in the school population, with a quick enrollment decline from about 68,000 to 32,000 students—slowly climbing back to 42,000 by 2011. The outward migration of students resulted in a much smaller proportion of children in poverty in the city. Additionally, funds were reduced for traditional public schools while funds for charter schools continued to be provided through federal, state and foundation sources.

Drawing an appropriate comparison group and making well-founded conclusions becomes exceptionally problematic in a city with such fundamental changes and such potentially strong selection effects. This is particularly true when 78% of New Orleans public school students are enrolled in charter schools.

The new study’s findings are also problematic because of weaknesses in the study itself. In reviewing the larger national study for the Think Twice think tank review project, Andrew Maul and Abby McClelland found “significant reasons for caution in interpreting the study’s results.” As the methods used for Louisiana are virtually identical, these same concerns carry forward. Among other things, CREDO’s researchers do not sufficiently justify their estimation of growth, which they continue to express in the problematic “days of learning” metric. As well, they use regression models that fail to address independence of observations and the absence of measurement error – two key assumptions required in such analyses.

Even setting aside these concerns, the effect sizes reported for New Orleans—let alone for the state as a whole—are not impressive in terms of absolute magnitude. Differences of 0.12 standard deviations in reading and 0.14 in mathematics indicate that less than one half of one percent of the variation in test scores is explanable by membership in a charter school.

The study’s methods raise concerns that the findings could easily be misinterpreted to inflate pro-charter conclusions. In context, there’s little to crow about: the results from Louisiana and New Orleans are not much different from the uninspiring national results; the results for the state’s suburban charter schools showed negative gain scores (somewhat less growth in charters than in the comparison schools); and the small positive results reported for New Orleans are confounded by the devastating aftermath of a unique disaster.

There are undoubtedly charter schools in New Orleans that are providing high-quality educational opportunities, and they deserve praise. But the evidence provided by the new CREDO study falls well short of Gov. Jindal’s “proof” of the success of charter schools in either the city or the state.

The Think Twice think tank review project () of the National Education Policy Center (NEPC) provides the public, policy makers, and the press with timely, academically sound reviews of selected publications. NEPC is housed at the University of Colorado Boulder School of Education. The Think Twice think tank review project is made possible in part by support provided by the Great Lakes Center for Education Research and Practice.