AEI’s Attempt to Discover What Worked with NCLB Is Laudable but Flawed

Does the threat of sanctions from the federal No Child Left Behind (NCLB) act lead schools to boost achievement scores? A recent study from the American Enterprise Institute (AEI) suggests the answer to that question is a cautious “yes.”

A new review released today, however, says that the report does not succeed in sorting out the effects of different elements of NCLB. It finds tepid effects from just one element of the federal law.

The report from AEI is titled, Were all those standardized tests for nothing? The Lessons of No Child Left Behind, and was written for the free-market think tank by Thomas Ahn and Jacob Vigdor.

It was reviewed for the Think Twice think tank review project by Professor Bruce Fuller of the University of California, Berkeley. The review is published by the National Education Policy Center, housed at the University of Colorado Boulder School of Education. Fuller is a professor of education and public policy who studies how policy or civic action shapes the social organization of schools and student achievement.

The AEI report attempts to isolate the effects of NCLB’s threat of sanctions placed on underperforming schools with an examination of the experience of one state, North Carolina. NCLB was up for renewal in 2007, but Congress has failed to consider how the federal role in education might be changed, and the Obama administration’s education agenda has, as Fuller puts it, “largely moved past NCLB.” Moreover, House Republicans “passed legislation this summer that would detonate any strong federal presence, trashing several of the No Child ingredients that Ahn and Vigdor are earnestly and carefully trying to weigh.”

Fuller finds that the report does make “a helpful contribution” in unpacking policy levers that were part of NCLB and in identifying elements of the law that should receive empirical examination. Such an examination might establish the levers that might motivate students or teachers to improve school outcomes.

The AEI authors, he writes, are “generally skeptical” that NCLB has made a difference in improving achievement. At the same time, however, “the report does ascribe discrete yet small achievement gains to the sanction provisions of the law.”

The report, he notes, rests its conclusions on a brief literature review and on the AEI authors’ study of North Carolina achievement test scores. The N.C. study suggests, according to its authors, that “federal pressure and punishment are promising policy avenues.” Yet with an effect size of just 0.05 of a standard deviation attributed to NCLB sanctions, the effects of federal sanctions would be extremely small, even if we assume no collateral effects from North Carolina’s own policy efforts, including cash incentives to schools demonstrating real progress.

In fact, notwithstanding the AEI authors’ assertion “that NCLB’s specific policy levers can be definitively estimated amidst all the collateral policy noise,” Fuller expresses reasons for doubt. He also notes that “the reader learns little about how local educators comprehend or respond to federal and state accountability pressures.” Accordingly, the report “does not help us understand what elements of NCLB have lifted students and which have failed.” He concludes, “Readers of the AEI report will benefit from the information provided, but they will remain uninformed as to how the search for an effective federal role can be informed by scarce evidence on the effects of various elements of NCLB.”