Philadelphia’s Renaissance Schools at 18 Months

Ω

Can chronically low-performing schools dramatically improve in a short period of time? That was the question that the Renaissance Schools Initiative – Philadelphia’s approach to the turnaround school reform model – sought to answer when it was implemented in 2009.

Eighteen months into the Initiative, as the School District of Philadelphia and the School Reform Commission deliberate its future against the backdrop of severe budget cuts, Research for Action (RFA) has released results of its evaluation of the Renaissance Schools.

RFA’s research represents an exhaustive study of school turnarounds– a key element in federal and state education reforms. The study focused on determining whether the first group of 13 schools – both District-run Promise Academies and Charter-managed schools – made early progress toward the longer-term goal of dramatically improving student outcomes.

The Institute of Education Sciences/ What Works Clearinghouse released the following Quick Review:

What is the study about?
The study examined the effectiveness of Philadelphia’s Renaissance Schools Initiative after one year of implementation. The Renaissance Schools Initiative, which began in the 2010–11 school year, aimed at improving low-performing schools by providing new management, additional resources, and new educational strategies.

What did the study report?
The study reported that students in grade K–8 Renaissance Schools had higher math achievement, reading achievement, and attendance rates than students in comparison schools.

How does the WWC rate this study?
This study does not meet WWC evidence standards because the Renaissance schools and comparison schools did not have similar achievement levels in the year before the Renaissance Schools Initiative began. Therefore, any changes in student achievement or attendance cannot be attributed solely to the implementation of the Renaissance Schools Initiative.

Citation
Gold, E., Norton, M. H., Good, D., & Levin, S. (2012). Philadelphia’s Renaissance Schools Initiative: 18 Month Interim Report. Philadelphia, PA: Research for Action.

In response RFA released the following:

Statement on the Institute of Education Sciences/What Works Clearinghouse Rating of Renaissance Schools Initiative: 18 Month Interim Report

The What Works Clearinghouse (WWC) yesterday released a rating of Research for Action’s most recent evaluation of Philadelphia’s Renaissance Schools Initiative. The rating – does not meet WWC’s evidence standards – was assigned with the explanation that “the Renaissance schools and comparison schools did not have similar achievement levels in the year before the Renaissance Schools Initiative began. Therefore, any changes in student achievement or attendance cannot be attributed solely to the implementation of the Renaissance Schools Initiative.”

However, further explanation is required to clarify the WWC’s rating. It was not possible to create a comparison school group with equivalent achievement levels prior to the Renaissance Initiative because the District selected the lowest performing schools in the District to participate in the reform effort, thereby removing the possibility of identifying a fully equivalent comparison school group. In identifying schools to participate in the Initiative, the School District of Philadelphia included all of the lowest-performing schools in the District, based on the District’s School Performance Index (SPI). As a result, researchers identified a set of comparison schools that most clearly mirrored the Renaissance Schools.

RFA utilized two sets of controls to rule out alternative explanations of the performance of the Renaissance Schools. First, the Renaissance Schools were compared to 72 schools in the District that were as similar as possible to the Renaissance Schools: They had very low School Performance Indices, and very similar demographic characteristics. Second, RFA utilized an interrupted time series design that compared the performance of both the Renaissance Schools and the comparison schools five years prior to the Renaissance Schools Initiative, and then one year after, to determine whether school performance differed prior to the Initiative. RFA’s analyses revealed that the rate of student growth in the five years prior to the start of the Renaissance Initiative was statistically equivalent to the comparison group of schools. Given the implementation of the Initiative, this research design provides the most rigorous examination possible of its impact. As such, RFA’s study provides strong evidence of an early, positive effect of the Renaissance Schools reform model.

Kate Shaw, RFA’s executive director, said in response to the WWC’s rating: “The District’s goal with the Renaissance Schools Initiative was to significantly improve student performance in the lowest performing schools in the district—not to conduct a scientific experimental study by randomly assigning schools to the Initiative. RFA constructed the most rigorous study available given the lack of random assignment. Our analyses detected initial, encouraging gains in student achievement and attendance in all K-8 Renaissance schools, and it is highly likely that these gains are due to participation in the Renaissance Schools Initiative. However, we continue to feel strongly that more research is needed to determine whether these gains will be sustained over time. RFA remains committed to its goal of contributing the rigorous research needed to make responsible decisions to the field of education, and we stand behind the integrity and accuracy of our evaluation on the Renaissance Schools.”

Ω