Follow-up on Effect of New Orleans School Reforms on Student Achievement,

In 2005, Hurricane Katrina forever changed the city of New Orleans and its schools. In the wake of the deadly storm, the educational system underwent a series of dramatic and controversial transformations. Nearly all the schools were converted from traditional public institutions to publicly-funded, independently-operated charters. Families selected schools rather than being assigned to buildings close to their homes. The district dismissed all of its employees, and many never returned. Per-pupil funding increased by 10 to 15 percent. 

How did all of this change impact key educational outcomes?

In July, National Education Policy Fellow Douglas N. Harris and his co-author Matthew F. Larsen addressed this question in What Effect Did the New Orleans School Reforms Have on Student Achievement, High School Graduation, and College Outcomes?, a policy brief and accompanying technical report published by the Education Research Alliance for New Orleans at Tulane University.

The study found that student achievement, high school graduation rates and college outcomes all improved in the wake of the reforms. In the weeks since its publication, New York Times columnist David Leonhardt has cited the study in support of charter schools, while charter skeptics have raised objections. 

In the Q&A below, Harris discusses the study and its results. 

Q: NEPC has done a lot of work focused on closing opportunity gaps, based on the foundational premise that students learn more when they have greater opportunities to learn – and that achievement gaps are the direct result of opportunity gaps. Those opportunity gaps arise from many different sources, inside of schools and outside of schools. Have you and your colleagues found that the post-Katrina schools in New Orleans are increasing opportunities to learn (OTL) in concrete ways? If so, what are some examples?

Harris: OTL usually includes things like instructional quality, curriculum, class sizes, and school climate. In New Orleans, the student-teacher ratio is essentially unchanged. Teacher experience is lower. On other measures, perhaps the best evidence we have comes from a survey of teachers who taught in New Orleans schools both pre- and post-Katrina. The results present a mixed picture. School climate and support for teachers improved, but teachers are less satisfied with their jobs and with the evaluation process. 

We don’t want to stop there, however, because OTL is hard to measure and because, while OTL measures are positively related to student outcomes, their effects on student outcomes are generally small. Therefore, it’s also worth thinking about outcomes and achievement gaps—whether opportunity to learn turns into measurable learning. We found that all racial/ethnic and income groups saw increased outcomes on every measure in New Orleans—test scores, high school graduation, college entry, college persistence, and college graduation. Also, achievement gaps within the city declined or remaining unchanged on almost all of these measures. 

Q: Charter school research from next door in Texas, conducted by Will Dobbie and Roland Fryer Jr., found that later earnings outcomes are actually worse for students in charters, even when test-score outcomes improve. Outcome measures for charters are better in places like Boston. More generally, research measuring differences in outcomes between charter schools and public schools suggests that such differences are non-existent or very small. Do you have a hypothesis or a hunch as to why NOLA’s results would be better? Are the examples that you cite, things that NOLA is doing differently than, for example, Texas charters?

Harris:The differences in outcomes between charter and traditional public schools are indeed small. In the early years, charters looked slightly worse on average, and now they look slightly better. An important question going forward is whether that upward trend will continue.

One clear pattern in the research is that “no excuses” schools seem to have more positive effects on typical student outcome measures than other kinds of charter schools. This is true in Boston as well as in the Dobbie and Fryer study. (Actually, the pattern with no excuses also aligns with the old effective schools literature.) New Orleans, too, has had a large share of schools that might be described as no excuses.

No excuses schools also tend to spend more money, and we do see higher spending in New Orleans. It may be the combination of schooling model and spending. 

Q: The technical report discusses six possible threats to validity in the study, one of which concerns the question of how much of the measured outcomes might be due to increased resources. The school-choice reforms were accompanied by an increase of almost $1,400 annually per student. You’ve noted that this increase in spending likely contributed some to the overall outcome improvement that you found. But you also noted that the increased funding would not have been forthcoming if the district didn’t adopt the school-choice reforms. So you said that it’s hard to interpret the funding increase as a cause that’s separate or alternative from the school-choice reforms. This leads to several questions: How do we know how much (if any) of the outcome improvements was due to resources and how much (if any) is due to school choice? Given the research from Kirabo Jackson and his colleagues, finding that a 20 percent increase in school spending led to 25 percent higher earnings and a 20 percentage-point reduction in the incidence of adult poverty, can you estimate how much of the outcome improvements you’re seeing in NOLA might be due to, e.g., school-choice reforms and how much due to the increased spending? I.e., what percentage of the variability is explained by school funding increases versus other factors?

Harris: First, it’s important to recognize the important contributions of the studies by Jackson, Johnson, and Persico and Lafortune, and Lafortune, Rothstein, and Schanzenbach in identifying causal effects of school spending. Money matters, and that’s why we carried out the study showing the increase in funding that you’re referring to.

It is difficult to estimate the role of funding or really any specific factor since this was a system-level change, involving several interconnected factors. Certainly, getting something as precise as “the percentage of variability” is impossible. Even if we could, there is the other issue you raised—that the increased spending was partly caused by the reforms.

Q: What lessons do you think the NOLA reform offers for places that don’t include the hefty funding increase?

Harris:The effects would almost certainly be smaller, but it’s hard to tell by how much. A related question is, what would have happened if we had kept the old system, but still increased funding? The answer is probably “not much.” Even today’s critics of the charter-based reforms say that the district was in need of an overhaul. Pouring money into a failing district isn’t the answer, nor is it politically plausible in the long run. 

Q: A columnist for the New York Times has been using your study as part of his advocacy for charter schools. This has drawn some responses, as you know. To be fair to the columnist, an op-ed isn’t amenable to a discussion of the potential limitations mentioned in the technical report. But to what extent, if any, is it appropriate to draw upon your study to advocate for charter schools in settings outside of New Orleans? To what extent, if any, have the New York Times columns extrapolated lessons that are beyond the scope of your findings? 

Harris: In some sense, any policy advocacy based on research requires some degree of extrapolation. Certainly, that’s true here as well. As we emphasized in the summary of our briefing paper, New Orleans was uniquely situated for these reforms to work. The district was extremely low-performing and pretty much everyone agreed that some type of major change was in order. It’s easier to improve from such a low starting point. Also, the national interest in rebuilding the city and being part of the reform effort made it easier to attract educators, especially in the early years. Cities tend to have advantages over suburbs and rural areas as well. In short, I don’t think we can extrapolate New Orleans to most of the country. It’s more like a best-case scenario.

Q: From outside of New Orleans, it seems like a great deal has been changing not just in the city’s schools, but also in its communities—which have had to be rebuilt as well. NEPC’s opportunity-gap work has been strongly influenced by the body of research showing that outside-of-school difference in resources and opportunities are substantially more important for kids’ educational outcomes than are inside-of-school differences. What sorts of changes, if any, have you and your colleagues seen in New Orleans since the reforms that might be important? Are those differences ones that researchers can account for?

Harris: Researchers usually break these “outside-of-school” factors into two categories: home and community. The family income and parent education of individual students, in particular, are strong predictors of education outcomes. I think you’re coming to a question about population change later. 

On community factors, New Orleans, unfortunately, has a long history of violence, mass incarceration, racism, and deeply impoverished neighborhoods. We are currently studying the possibility of school reform effects on crime. All we know at this point, however, is that crime and incarceration rates are somewhat lower in New Orleans compared with pre-Katrina, but this is also true statewide. So, while we see no obvious indications of relative improvement on crime and incarceration, as indicators of progress in the community, we can’t draw any conclusions about them at this point. 

Q:How, if at all, did your study account for any changes in the rate of concentrated poverty at the school level pre- versus post-Katrina?

Harris: The New Orleans reforms affected the whole system and, as we’ve discussed, poverty and demographics in the public schools were essentially unchanged. In other words, it’s really the overall public school demographics in the city that could affect our results. Since those demographics changed only very slightly, it’s hard to see how this could influence the results.

But the concentration of poverty by school is important for other reasons and we’ve studied that, too. In an earlier study, with Lindsay Bell Weixler, Nathan Barrett, and Jennifer Jennings, we focused on cross-school segregation across a wide range of student characteristics—race, income, special education, English Language Learners, and achievement. We found, first, that the schools were very segregated prior to the reforms. After the reforms, we found no systematic changes in segregation in these measures at the elementary level. At the high school level, we see increased segregation on poverty and race, but decreased segregation on achievement and special education.

Q:  You and your colleagues clearly attempted to control for pre- and post-Katrina changes in population. To what extent do you think that the demographic variables available to you as a researcher can account for the change in the composition of the student population that occurred pre- versus post-Katrina? To what extent is it the case that the pre-Katrina student population was substantially more disadvantaged than the post-Katrina population?

Harris: The New Orleans population was disadvantaged before and that stayed about the same. We’re confident about that because we come to the same conclusion from three entirely different types of analysis. First, we looked at the share of students eligible for free or reduced prices lunches. Second, we commissioned the U.S. Census Bureau to provide detailed family data on public school students, before and after the reforms. Third, we looked at the pre-Katrina scores of students who returned to the city and compared those with students who did not return. All three analyses suggest the demographics changed very little, so little that they can’t have more than a negligible effect on student outcomes. Some of the analyses even suggest that students became more disadvantaged after the reforms, indicating that we might be slightly understating the reform effects.

Just to clarify, the population of the city has definitely changed, but the demographics of the city don’t mirror the population of the public schools. Many families don’t have children, and New Orleans sends about 25 percent of school-age children to private schools (that hasn’t changed much either). The overall city demographics therefore aren’t very informative. 

Q: College enrollment/persistence/completion rates are probably not impacted by gaming or accountability in K-12. But to what extent might the improvements in test scores and high school graduation rates post-Katrina New Orleans be attributable to gaming incentivized by high-stakes accountability?

Harris: Gaming the system on tests, such as drilling test questions, is the hardest to gauge. There’s no question that New Orleans schools are data-driven and under tremendous pressure to get scores up, so there is probably some of that going on. On the other hand, Louisiana has aggressive accountability statewide, so it could be that other districts, which comprise our comparison groups, may also be doing this, so some of it washes out when we estimate the effects. Unfortunately, we can’t use the NAEP to help us out here. 

With high school graduation, the potential for gaming is there, but we were able to test for it directly—and we don’t see any evidence of it. And, as you say, this isn’t an issue with college graduation.

Q: I wonder if you could tell us how we should think about the internal validity of the study (i.e., how should a reader define and bound the intervention, and how sure are you that you’re measuring that intervention?). Similarly, how should readers think about the external validity of the study? To what extent should someone in Memphis or Minneapolis or Modesto assume that they would get similar results if they followed the New Orleans playbook, and what are the key elements of that playbook?

Harris: Overall, I would say that internal validity is stronger than external validity. With internal validity, the main complicating factor is the role of funding. One option is to just think of the increased spending as part of the reform “package.” The other is to try and isolate the two, but as we discussed earlier, that’s complicated by the fact that the reforms partly caused the funding increase. 

On external validity, we have good reasons to expect that market-based reforms work better in urban areas and that any reform will tend to generate more improvement when the existing system is failing—in the sense of generating little outcome growth or value-added. So, this type of reform, and others, are more likely to work in the very lowest performing urban districts in the country. Also, having a lot of universities and college-educated workers will make it easier to attract educators who fit the educational models that charter schools want to create. 

It’s important to note that these conditions aren’t very common nationally. Again, New Orleans is probably a best-case scenario in terms of student outcomes.