Review Worth Sharing Identifies Flaws in NC Voucher Evaluation

An evaluation of an education program typically gives some information about whether or not a program is working. But a recent evaluation of North Carolina’s school voucher program is so flawed methodologically that it fails to explain whether the state’s Opportunity Scholarships help or harm a student’s education, according to a review by Kris Nordstrom, an education policy consultant on the Education and Law Project at the North Carolina Justice Center, a social justice-focused research and advocacy organization. Nordstrom’s review is part of a new NEPC feature called Reviews Worth Sharing, which are not commissioned or edited by NEPC but that we believe contribute to our goal of helping policymakers, reporters, and others assess the social science merit of reports and judge their value in guiding policy. The views and conclusions addressed belong entirely to the author. The evaluation reviewed, An Impact Analysis of North Carolina’s Opportunity Scholarship Program on Student Achievement, is a working paper by North Carolina State researchers Anna J. Egalite, D.T. Stallings, and Stephen R. Porter. The review finds that methodological flaws in the evaluation make it impossible to accurately compare North Carolina private school students who receive the vouchers with their public school counterparts who do not. It is also possible that the private school students who participated in the analysis were not representative of the average voucher student. That’s because the working paper only examined a small, non-random handful of voucher students (89 individuals, or 1.6 percent of all voucher recipients) who volunteered to be tested for the evaluation. In addition, just over half of the private schools attended by these 89 recipients were Catholic. Yet only 10 percent of all North Carolina voucher schools are Catholic. The evaluation did use a statistical method called propensity-score matching to create a public school comparison group that was designed to be similar to the pool of private school volunteers. However, Nordstrom identifies five main flaws with this comparison: The private school students who volunteered to participate in the evaluation were recruited by a pro-voucher advocacy organization, Parents for Educational Freedom in North Carolina. The evaluation does not clarify to what extent, if any, the organization cherry-picked the volunteers or their schools. The public school students likely came from lower-income families than the voucher recipients. Evaluation authors said that they accounted for this difference by incorporating prior year’s test results into the analysis. But that assumes that income differences did not impact performance in the ensuing school year.  The public school students likely attended schools with higher poverty rates than the private school students would have been attending, absent the vouchers. Again, evaluation authors said that they accounted for this difference by incorporating prior year’s test results into the analysis, but that (again) assumes that the differences did not impact performance in the ensuing school year. It is possible that the public and private school students had different levels of motivation when taking the test. While voucher recipients might have perceived that their performance could impact their ability to remain in their private schools, the public school students likely viewed the exam as a meaningless exercise. The test used in the evaluation was not aligned to North Carolina’s Standard Course of Study. If it was aligned more closely with the private schools’ curricula, that could give the voucher recipients an advantage. North Carolina’s voucher program is scheduled to grow by $10 million per year, to $144.8 million in 2027-28. Yet as Nordstrom concludes:

North Carolina General Assembly lawmakers are about to conclude yet another legislative session without implementing meaningful evaluation and accountability measures on state voucher programs. Despite the N.C. State report, unfettered expansion of vouchers continues, and policymakers, educators, and parents still don’t know whether the program is working or not.